This is a major release from 0.12.0 and includes a number of API changes, several new features and enhancements along with a large number of bug fixes.
Highlights include:
support for a new index type Float64Index, and other Indexing enhancements
Float64Index
HDFStore has a new string based syntax for query specification
HDFStore
support for new methods of interpolation
updated timedelta operations
timedelta
a new string manipulation method extract
extract
Nanosecond support for Offsets
isin for DataFrames
isin
Several experimental features are added, including:
new eval/query methods for expression evaluation
eval/query
support for msgpack serialization
msgpack
an i/o interface to Google’s BigQuery
BigQuery
Their are several new or updated docs sections including:
Comparison with SQL, which should be useful for those familiar with SQL but still learning pandas.
Comparison with R, idiom translations from R to pandas.
Enhancing Performance, ways to enhance pandas performance with eval/query.
Warning
In 0.13.0 Series has internally been refactored to no longer sub-class ndarray but instead subclass NDFrame, similar to the rest of the pandas containers. This should be a transparent change with only very limited API implications. See Internal Refactoring
Series
ndarray
NDFrame
read_excel now supports an integer in its sheetname argument giving the index of the sheet to read in (GH4301).
read_excel
sheetname
Text parser now treats anything that reads like inf (“inf”, “Inf”, “-Inf”, “iNf”, etc.) as infinity. (GH4220, GH4219), affecting read_table, read_csv, etc.
read_table
read_csv
pandas now is Python 2/3 compatible without the need for 2to3 thanks to @jtratner. As a result, pandas now uses iterators more extensively. This also led to the introduction of substantive parts of the Benjamin Peterson’s six library into compat. (GH4384, GH4375, GH4372)
pandas
six
pandas.util.compat and pandas.util.py3compat have been merged into pandas.compat. pandas.compat now includes many functions allowing 2/3 compatibility. It contains both list and iterator versions of range, filter, map and zip, plus other necessary elements for Python 3 compatibility. lmap, lzip, lrange and lfilter all produce lists instead of iterators, for compatibility with numpy, subscripting and pandas constructors.(GH4384, GH4375, GH4372)
pandas.util.compat
pandas.util.py3compat
pandas.compat
lmap
lzip
lrange
lfilter
numpy
Series.get with negative indexers now returns the same as [] (GH4390)
Series.get
[]
Changes to how Index and MultiIndex handle metadata (levels, labels, and names) (GH4039):
Index
MultiIndex
levels
labels
names
# previously, you would have set levels or labels directly >>> pd.index.levels = [[1, 2, 3, 4], [1, 2, 4, 4]] # now, you use the set_levels or set_labels methods >>> index = pd.index.set_levels([[1, 2, 3, 4], [1, 2, 4, 4]]) # similarly, for names, you can rename the object # but setting names is not deprecated >>> index = pd.index.set_names(["bob", "cranberry"]) # and all methods take an inplace kwarg - but return None >>> pd.index.set_names(["bob", "cranberry"], inplace=True)
All division with NDFrame objects is now truedivision, regardless of the future import. This means that operating on pandas objects will by default use floating point division, and return a floating point dtype. You can use // and floordiv to do integer division.
//
floordiv
Integer division
In [3]: arr = np.array([1, 2, 3, 4]) In [4]: arr2 = np.array([5, 3, 2, 1]) In [5]: arr / arr2 Out[5]: array([0, 0, 1, 4]) In [6]: pd.Series(arr) // pd.Series(arr2) Out[6]: 0 0 1 0 2 1 3 4 dtype: int64
True Division
In [7]: pd.Series(arr) / pd.Series(arr2) # no future import required Out[7]: 0 0.200000 1 0.666667 2 1.500000 3 4.000000 dtype: float64
Infer and downcast dtype if downcast='infer' is passed to fillna/ffill/bfill (GH4604)
downcast='infer'
fillna/ffill/bfill
__nonzero__ for all NDFrame objects, will now raise a ValueError, this reverts back to (GH1073, GH4633) behavior. See gotchas for a more detailed discussion.
__nonzero__
ValueError
This prevents doing boolean comparison on entire pandas objects, which is inherently ambiguous. These all will raise a ValueError.
>>> df = pd.DataFrame({'A': np.random.randn(10), ... 'B': np.random.randn(10), ... 'C': pd.date_range('20130101', periods=10) ... }) ... >>> if df: ... pass ... Traceback (most recent call last): ... ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all(). >>> df1 = df >>> df2 = df >>> df1 and df2 Traceback (most recent call last): ... ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all(). >>> d = [1, 2, 3] >>> s1 = pd.Series(d) >>> s2 = pd.Series(d) >>> s1 and s2 Traceback (most recent call last): ... ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().
Added the .bool() method to NDFrame objects to facilitate evaluating of single-element boolean Series:
.bool()
In [1]: pd.Series([True]).bool() Out[1]: True In [2]: pd.Series([False]).bool() Out[2]: False In [3]: pd.DataFrame([[True]]).bool() Out[3]: True In [4]: pd.DataFrame([[False]]).bool() Out[4]: False
All non-Index NDFrames (Series, DataFrame, Panel, Panel4D, SparsePanel, etc.), now support the entire set of arithmetic operators and arithmetic flex methods (add, sub, mul, etc.). SparsePanel does not support pow or mod with non-scalars. (GH3765)
DataFrame
Panel
Panel4D
SparsePanel
pow
mod
Series and DataFrame now have a mode() method to calculate the statistical mode(s) by axis/Series. (GH5367)
mode()
Chained assignment will now by default warn if the user is assigning to a copy. This can be changed with the option mode.chained_assignment, allowed options are raise/warn/None. See the docs.
mode.chained_assignment
raise/warn/None
In [5]: dfc = pd.DataFrame({'A': ['aaa', 'bbb', 'ccc'], 'B': [1, 2, 3]}) In [6]: pd.set_option('chained_assignment', 'warn')
The following warning / exception will show if this is attempted.
In [7]: dfc.loc[0]['A'] = 1111
Traceback (most recent call last) ... SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_index,col_indexer] = value instead
Here is the correct method of assignment.
In [8]: dfc.loc[0, 'A'] = 11 In [9]: dfc Out[9]: A B 0 11 1 1 bbb 2 2 ccc 3
Panel.reindex
Panel.reindex(items=None, major_axis=None, minor_axis=None, **kwargs)
to conform with other NDFrame objects. See Internal Refactoring for more information.
Series.argmin
Series.argmax
Series.idxmin
Series.idxmax
min or max element respectively. Prior to 0.13.0 these would return the position of the min / max element. (GH6214)
These were announced changes in 0.12 or prior that are taking effect as of 0.13.0
Remove deprecated Factor (GH3650)
Factor
Remove deprecated set_printoptions/reset_printoptions (GH3046)
set_printoptions/reset_printoptions
Remove deprecated _verbose_info (GH3215)
_verbose_info
Remove deprecated read_clipboard/to_clipboard/ExcelFile/ExcelWriter from pandas.io.parsers (GH3717) These are available as functions in the main pandas namespace (e.g. pd.read_clipboard)
read_clipboard/to_clipboard/ExcelFile/ExcelWriter
pandas.io.parsers
pd.read_clipboard
default for tupleize_cols is now False for both to_csv and read_csv. Fair warning in 0.12 (GH3604)
tupleize_cols
False
to_csv
default for display.max_seq_len is now 100 rather then None. This activates truncated display (“…”) of long sequences in various places. (GH3391)
Deprecated in 0.13.0
deprecated iterkv, which will be removed in a future release (this was an alias of iteritems used to bypass 2to3’s changes). (GH4384, GH4375, GH4372)
iterkv
2to3
deprecated the string method match, whose role is now performed more idiomatically by extract. In a future release, the default behavior of match will change to become analogous to contains, which returns a boolean indexer. (Their distinction is strictness: match relies on re.match while contains relies on re.search.) In this release, the deprecated behavior is the default, but the new behavior is available through the keyword argument as_indexer=True.
match
contains
re.match
re.search
as_indexer=True
Prior to 0.13, it was impossible to use a label indexer (.loc/.ix) to set a value that was not contained in the index of a particular axis. (GH2578). See the docs
.loc/.ix
In the Series case this is effectively an appending operation
In [10]: s = pd.Series([1, 2, 3]) In [11]: s Out[11]: 0 1 1 2 2 3 dtype: int64 In [12]: s[5] = 5. In [13]: s Out[13]: 0 1.0 1 2.0 2 3.0 5 5.0 dtype: float64
In [14]: dfi = pd.DataFrame(np.arange(6).reshape(3, 2), ....: columns=['A', 'B']) ....: In [15]: dfi Out[15]: A B 0 0 1 1 2 3 2 4 5
This would previously KeyError
KeyError
In [16]: dfi.loc[:, 'C'] = dfi.loc[:, 'A'] In [17]: dfi Out[17]: A B C 0 0 1 0 1 2 3 2 2 4 5 4
This is like an append operation.
append
In [18]: dfi.loc[3] = 5 In [19]: dfi Out[19]: A B C 0 0 1 0 1 2 3 2 2 4 5 4 3 5 5 5
A Panel setting operation on an arbitrary axis aligns the input to the Panel
In [20]: p = pd.Panel(np.arange(16).reshape(2, 4, 2), ....: items=['Item1', 'Item2'], ....: major_axis=pd.date_range('2001/1/12', periods=4), ....: minor_axis=['A', 'B'], dtype='float64') ....: In [21]: p Out[21]: <class 'pandas.core.panel.Panel'> Dimensions: 2 (items) x 4 (major_axis) x 2 (minor_axis) Items axis: Item1 to Item2 Major_axis axis: 2001-01-12 00:00:00 to 2001-01-15 00:00:00 Minor_axis axis: A to B In [22]: p.loc[:, :, 'C'] = pd.Series([30, 32], index=p.items) In [23]: p Out[23]: <class 'pandas.core.panel.Panel'> Dimensions: 2 (items) x 4 (major_axis) x 3 (minor_axis) Items axis: Item1 to Item2 Major_axis axis: 2001-01-12 00:00:00 to 2001-01-15 00:00:00 Minor_axis axis: A to C In [24]: p.loc[:, :, 'C'] Out[24]: Item1 Item2 2001-01-12 30.0 32.0 2001-01-13 30.0 32.0 2001-01-14 30.0 32.0 2001-01-15 30.0 32.0
Added a new index type, Float64Index. This will be automatically created when passing floating values in index creation. This enables a pure label-based slicing paradigm that makes [],ix,loc for scalar indexing and slicing work exactly the same. See the docs, (GH263)
[],ix,loc
Construction is by default for floating type values.
In [20]: index = pd.Index([1.5, 2, 3, 4.5, 5]) In [21]: index Out[21]: Float64Index([1.5, 2.0, 3.0, 4.5, 5.0], dtype='float64') In [22]: s = pd.Series(range(5), index=index) In [23]: s Out[23]: 1.5 0 2.0 1 3.0 2 4.5 3 5.0 4 dtype: int64
Scalar selection for [],.ix,.loc will always be label based. An integer will match an equal float index (e.g. 3 is equivalent to 3.0)
[],.ix,.loc
3
3.0
In [24]: s[3] Out[24]: 2 In [25]: s.loc[3] Out[25]: 2
The only positional indexing is via iloc
iloc
In [26]: s.iloc[3] Out[26]: 3
A scalar index that is not found will raise KeyError
Slicing is ALWAYS on the values of the index, for [],ix,loc and ALWAYS positional with iloc
In [27]: s[2:4] Out[27]: 2.0 1 3.0 2 dtype: int64 In [28]: s.loc[2:4] Out[28]: 2.0 1 3.0 2 dtype: int64 In [29]: s.iloc[2:4] Out[29]: 3.0 2 4.5 3 dtype: int64
In float indexes, slicing using floats are allowed
In [30]: s[2.1:4.6] Out[30]: 3.0 2 4.5 3 dtype: int64 In [31]: s.loc[2.1:4.6] Out[31]: 3.0 2 4.5 3 dtype: int64
Indexing on other index types are preserved (and positional fallback for [],ix), with the exception, that floating point slicing on indexes on non Float64Index will now raise a TypeError.
[],ix
TypeError
In [1]: pd.Series(range(5))[3.5] TypeError: the label [3.5] is not a proper indexer for this index type (Int64Index) In [1]: pd.Series(range(5))[3.5:4.5] TypeError: the slice start [3.5] is not a proper indexer for this index type (Int64Index)
Using a scalar float indexer will be deprecated in a future version, but is allowed for now.
In [3]: pd.Series(range(5))[3.0] Out[3]: 3
Query Format Changes. A much more string-like query format is now supported. See the docs.
In [32]: path = 'test.h5' In [33]: dfq = pd.DataFrame(np.random.randn(10, 4), ....: columns=list('ABCD'), ....: index=pd.date_range('20130101', periods=10)) ....: In [34]: dfq.to_hdf(path, 'dfq', format='table', data_columns=True) --------------------------------------------------------------------------- ImportError Traceback (most recent call last) <ipython-input-34-0aef61ba29c9> in <module> ----> 1 dfq.to_hdf(path, 'dfq', format='table', data_columns=True) ~/scipy/pandas/pandas/core/generic.py in to_hdf(self, path_or_buf, key, mode, complevel, complib, append, format, index, min_itemsize, nan_rep, dropna, data_columns, errors, encoding) 2468 data_columns=data_columns, 2469 errors=errors, -> 2470 encoding=encoding, 2471 ) 2472 ~/scipy/pandas/pandas/io/pytables.py in to_hdf(path_or_buf, key, value, mode, complevel, complib, append, format, index, min_itemsize, nan_rep, dropna, data_columns, errors, encoding) 278 if isinstance(path_or_buf, str): 279 with HDFStore( --> 280 path_or_buf, mode=mode, complevel=complevel, complib=complib 281 ) as store: 282 f(store) ~/scipy/pandas/pandas/io/pytables.py in __init__(self, path, mode, complevel, complib, fletcher32, **kwargs) 516 raise ValueError("format is not a defined argument for HDFStore") 517 --> 518 tables = import_optional_dependency("tables") 519 520 if complib is not None and complib not in tables.filters.all_complibs: ~/scipy/pandas/pandas/compat/_optional.py in import_optional_dependency(name, extra, raise_on_missing, on_version) 89 except ImportError: 90 if raise_on_missing: ---> 91 raise ImportError(msg) from None 92 else: 93 return None ImportError: Missing optional dependency 'tables'. Use pip or conda to install tables.
Use boolean expressions, with in-line function evaluation.
In [35]: pd.read_hdf(path, 'dfq', ....: where="index>Timestamp('20130104') & columns=['A', 'B']") ....: --------------------------------------------------------------------------- FileNotFoundError Traceback (most recent call last) <ipython-input-35-ca07b2dcdcd8> in <module> 1 pd.read_hdf(path, 'dfq', ----> 2 where="index>Timestamp('20130104') & columns=['A', 'B']") ~/scipy/pandas/pandas/io/pytables.py in read_hdf(path_or_buf, key, mode, errors, where, start, stop, columns, iterator, chunksize, **kwargs) 393 394 if not exists: --> 395 raise FileNotFoundError(f"File {path_or_buf} does not exist") 396 397 store = HDFStore(path_or_buf, mode=mode, errors=errors, **kwargs) FileNotFoundError: File test.h5 does not exist
Use an inline column reference
In [36]: pd.read_hdf(path, 'dfq', ....: where="A>0 or C>0") ....: --------------------------------------------------------------------------- FileNotFoundError Traceback (most recent call last) <ipython-input-36-9d864b636246> in <module> 1 pd.read_hdf(path, 'dfq', ----> 2 where="A>0 or C>0") ~/scipy/pandas/pandas/io/pytables.py in read_hdf(path_or_buf, key, mode, errors, where, start, stop, columns, iterator, chunksize, **kwargs) 393 394 if not exists: --> 395 raise FileNotFoundError(f"File {path_or_buf} does not exist") 396 397 store = HDFStore(path_or_buf, mode=mode, errors=errors, **kwargs) FileNotFoundError: File test.h5 does not exist
the format keyword now replaces the table keyword; allowed values are fixed(f) or table(t) the same defaults as prior < 0.13.0 remain, e.g. put implies fixed format and append implies table format. This default format can be set as an option by setting io.hdf.default_format.
format
table
fixed(f)
table(t)
put
fixed
io.hdf.default_format
In [37]: path = 'test.h5' In [38]: df = pd.DataFrame(np.random.randn(10, 2)) In [39]: df.to_hdf(path, 'df_table', format='table') --------------------------------------------------------------------------- ImportError Traceback (most recent call last) <ipython-input-39-172ee434571b> in <module> ----> 1 df.to_hdf(path, 'df_table', format='table') ~/scipy/pandas/pandas/core/generic.py in to_hdf(self, path_or_buf, key, mode, complevel, complib, append, format, index, min_itemsize, nan_rep, dropna, data_columns, errors, encoding) 2468 data_columns=data_columns, 2469 errors=errors, -> 2470 encoding=encoding, 2471 ) 2472 ~/scipy/pandas/pandas/io/pytables.py in to_hdf(path_or_buf, key, value, mode, complevel, complib, append, format, index, min_itemsize, nan_rep, dropna, data_columns, errors, encoding) 278 if isinstance(path_or_buf, str): 279 with HDFStore( --> 280 path_or_buf, mode=mode, complevel=complevel, complib=complib 281 ) as store: 282 f(store) ~/scipy/pandas/pandas/io/pytables.py in __init__(self, path, mode, complevel, complib, fletcher32, **kwargs) 516 raise ValueError("format is not a defined argument for HDFStore") 517 --> 518 tables = import_optional_dependency("tables") 519 520 if complib is not None and complib not in tables.filters.all_complibs: ~/scipy/pandas/pandas/compat/_optional.py in import_optional_dependency(name, extra, raise_on_missing, on_version) 89 except ImportError: 90 if raise_on_missing: ---> 91 raise ImportError(msg) from None 92 else: 93 return None ImportError: Missing optional dependency 'tables'. Use pip or conda to install tables. In [40]: df.to_hdf(path, 'df_table2', append=True) --------------------------------------------------------------------------- ImportError Traceback (most recent call last) <ipython-input-40-2e78d1aecbfa> in <module> ----> 1 df.to_hdf(path, 'df_table2', append=True) ~/scipy/pandas/pandas/core/generic.py in to_hdf(self, path_or_buf, key, mode, complevel, complib, append, format, index, min_itemsize, nan_rep, dropna, data_columns, errors, encoding) 2468 data_columns=data_columns, 2469 errors=errors, -> 2470 encoding=encoding, 2471 ) 2472 ~/scipy/pandas/pandas/io/pytables.py in to_hdf(path_or_buf, key, value, mode, complevel, complib, append, format, index, min_itemsize, nan_rep, dropna, data_columns, errors, encoding) 278 if isinstance(path_or_buf, str): 279 with HDFStore( --> 280 path_or_buf, mode=mode, complevel=complevel, complib=complib 281 ) as store: 282 f(store) ~/scipy/pandas/pandas/io/pytables.py in __init__(self, path, mode, complevel, complib, fletcher32, **kwargs) 516 raise ValueError("format is not a defined argument for HDFStore") 517 --> 518 tables = import_optional_dependency("tables") 519 520 if complib is not None and complib not in tables.filters.all_complibs: ~/scipy/pandas/pandas/compat/_optional.py in import_optional_dependency(name, extra, raise_on_missing, on_version) 89 except ImportError: 90 if raise_on_missing: ---> 91 raise ImportError(msg) from None 92 else: 93 return None ImportError: Missing optional dependency 'tables'. Use pip or conda to install tables. In [41]: df.to_hdf(path, 'df_fixed') --------------------------------------------------------------------------- ImportError Traceback (most recent call last) <ipython-input-41-b4473898f0cd> in <module> ----> 1 df.to_hdf(path, 'df_fixed') ~/scipy/pandas/pandas/core/generic.py in to_hdf(self, path_or_buf, key, mode, complevel, complib, append, format, index, min_itemsize, nan_rep, dropna, data_columns, errors, encoding) 2468 data_columns=data_columns, 2469 errors=errors, -> 2470 encoding=encoding, 2471 ) 2472 ~/scipy/pandas/pandas/io/pytables.py in to_hdf(path_or_buf, key, value, mode, complevel, complib, append, format, index, min_itemsize, nan_rep, dropna, data_columns, errors, encoding) 278 if isinstance(path_or_buf, str): 279 with HDFStore( --> 280 path_or_buf, mode=mode, complevel=complevel, complib=complib 281 ) as store: 282 f(store) ~/scipy/pandas/pandas/io/pytables.py in __init__(self, path, mode, complevel, complib, fletcher32, **kwargs) 516 raise ValueError("format is not a defined argument for HDFStore") 517 --> 518 tables = import_optional_dependency("tables") 519 520 if complib is not None and complib not in tables.filters.all_complibs: ~/scipy/pandas/pandas/compat/_optional.py in import_optional_dependency(name, extra, raise_on_missing, on_version) 89 except ImportError: 90 if raise_on_missing: ---> 91 raise ImportError(msg) from None 92 else: 93 return None ImportError: Missing optional dependency 'tables'. Use pip or conda to install tables. In [42]: with pd.HDFStore(path) as store: ....: print(store) ....: --------------------------------------------------------------------------- ImportError Traceback (most recent call last) <ipython-input-42-c3b5e2e57915> in <module> ----> 1 with pd.HDFStore(path) as store: 2 print(store) ~/scipy/pandas/pandas/io/pytables.py in __init__(self, path, mode, complevel, complib, fletcher32, **kwargs) 516 raise ValueError("format is not a defined argument for HDFStore") 517 --> 518 tables = import_optional_dependency("tables") 519 520 if complib is not None and complib not in tables.filters.all_complibs: ~/scipy/pandas/pandas/compat/_optional.py in import_optional_dependency(name, extra, raise_on_missing, on_version) 89 except ImportError: 90 if raise_on_missing: ---> 91 raise ImportError(msg) from None 92 else: 93 return None ImportError: Missing optional dependency 'tables'. Use pip or conda to install tables.
Significant table writing performance improvements
handle a passed Series in table format (GH4330)
can now serialize a timedelta64[ns] dtype in a table (GH3577), See the docs.
timedelta64[ns]
added an is_open property to indicate if the underlying file handle is_open; a closed store will now report ‘CLOSED’ when viewing the store (rather than raising an error) (GH4409)
is_open
a close of a HDFStore now will close that instance of the HDFStore but will only close the actual file if the ref count (by PyTables) w.r.t. all of the open handles are 0. Essentially you have a local instance of HDFStore referenced by a variable. Once you close it, it will report closed. Other references (to the same file) will continue to operate until they themselves are closed. Performing an action on a closed file will raise ClosedFileError
PyTables
ClosedFileError
In [43]: path = 'test.h5' In [44]: df = pd.DataFrame(np.random.randn(10, 2)) In [45]: store1 = pd.HDFStore(path) --------------------------------------------------------------------------- ImportError Traceback (most recent call last) <ipython-input-45-238c41f9cb75> in <module> ----> 1 store1 = pd.HDFStore(path) ~/scipy/pandas/pandas/io/pytables.py in __init__(self, path, mode, complevel, complib, fletcher32, **kwargs) 516 raise ValueError("format is not a defined argument for HDFStore") 517 --> 518 tables = import_optional_dependency("tables") 519 520 if complib is not None and complib not in tables.filters.all_complibs: ~/scipy/pandas/pandas/compat/_optional.py in import_optional_dependency(name, extra, raise_on_missing, on_version) 89 except ImportError: 90 if raise_on_missing: ---> 91 raise ImportError(msg) from None 92 else: 93 return None ImportError: Missing optional dependency 'tables'. Use pip or conda to install tables. In [46]: store2 = pd.HDFStore(path) --------------------------------------------------------------------------- ImportError Traceback (most recent call last) <ipython-input-46-c33bd0c968bb> in <module> ----> 1 store2 = pd.HDFStore(path) ~/scipy/pandas/pandas/io/pytables.py in __init__(self, path, mode, complevel, complib, fletcher32, **kwargs) 516 raise ValueError("format is not a defined argument for HDFStore") 517 --> 518 tables = import_optional_dependency("tables") 519 520 if complib is not None and complib not in tables.filters.all_complibs: ~/scipy/pandas/pandas/compat/_optional.py in import_optional_dependency(name, extra, raise_on_missing, on_version) 89 except ImportError: 90 if raise_on_missing: ---> 91 raise ImportError(msg) from None 92 else: 93 return None ImportError: Missing optional dependency 'tables'. Use pip or conda to install tables. In [47]: store1.append('df', df) --------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-47-2593530079f0> in <module> ----> 1 store1.append('df', df) NameError: name 'store1' is not defined In [48]: store2.append('df2', df) --------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-48-422371444bd8> in <module> ----> 1 store2.append('df2', df) NameError: name 'store2' is not defined In [49]: store1 --------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-49-b8e0fab9e793> in <module> ----> 1 store1 NameError: name 'store1' is not defined In [50]: store2 --------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-50-34781c1540e6> in <module> ----> 1 store2 NameError: name 'store2' is not defined In [51]: store1.close() --------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-51-1d8baa731449> in <module> ----> 1 store1.close() NameError: name 'store1' is not defined In [52]: store2 --------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-52-34781c1540e6> in <module> ----> 1 store2 NameError: name 'store2' is not defined In [53]: store2.close() --------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-53-6118cb252bbd> in <module> ----> 1 store2.close() NameError: name 'store2' is not defined In [54]: store2 --------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-54-34781c1540e6> in <module> ----> 1 store2 NameError: name 'store2' is not defined
removed the _quiet attribute, replace by a DuplicateWarning if retrieving duplicate rows from a table (GH4367)
_quiet
DuplicateWarning
removed the warn argument from open. Instead a PossibleDataLossError exception will be raised if you try to use mode='w' with an OPEN file handle (GH4367)
warn
open
PossibleDataLossError
mode='w'
allow a passed locations array or mask as a where condition (GH4467). See the docs for an example.
where
add the keyword dropna=True to append to change whether ALL nan rows are not written to the store (default is True, ALL nan rows are NOT written), also settable via the option io.hdf.dropna_table (GH4625)
dropna=True
True
io.hdf.dropna_table
pass through store creation arguments; can be used to support in-memory stores
The HTML and plain text representations of DataFrame now show a truncated view of the table once it exceeds a certain size, rather than switching to the short info view (GH4886, GH5550). This makes the representation more consistent as small DataFrames get larger.
To get the info view, call DataFrame.info(). If you prefer the info view as the repr for large DataFrames, you can set this by running set_option('display.large_repr', 'info').
DataFrame.info()
set_option('display.large_repr', 'info')
df.to_clipboard() learned a new excel keyword that let’s you paste df data directly into excel (enabled by default). (GH5070).
df.to_clipboard()
excel
read_html now raises a URLError instead of catching and raising a ValueError (GH4303, GH4305)
read_html
URLError
Added a test for read_clipboard() and to_clipboard() (GH4282)
read_clipboard()
to_clipboard()
Clipboard functionality now works with PySide (GH4282)
Added a more informative error message when plot arguments contain overlapping color and style arguments (GH4402)
to_dict now takes records as a possible out type. Returns an array of column-keyed dictionaries. (GH4936)
to_dict
records
NaN handing in get_dummies (GH4446) with dummy_na
NaN
# previously, nan was erroneously counted as 2 here # now it is not counted at all In [55]: pd.get_dummies([1, 2, np.nan]) Out[55]: 1.0 2.0 0 1 0 1 0 1 2 0 0 # unless requested In [56]: pd.get_dummies([1, 2, np.nan], dummy_na=True) Out[56]: 1.0 2.0 NaN 0 1 0 0 1 0 1 0 2 0 0 1
timedelta64[ns] operations. See the docs.
Most of these operations require numpy >= 1.7
numpy >= 1.7
Using the new top-level to_timedelta, you can convert a scalar or array from the standard timedelta format (produced by to_csv) into a timedelta type (np.timedelta64 in nanoseconds).
to_timedelta
np.timedelta64
nanoseconds
In [57]: pd.to_timedelta('1 days 06:05:01.00003') Out[57]: Timedelta('1 days 06:05:01.000030') In [58]: pd.to_timedelta('15.5us') Out[58]: Timedelta('0 days 00:00:00.000015') In [59]: pd.to_timedelta(['1 days 06:05:01.00003', '15.5us', 'nan']) Out[59]: TimedeltaIndex(['1 days 06:05:01.000030', '0 days 00:00:00.000015', NaT], dtype='timedelta64[ns]', freq=None) In [60]: pd.to_timedelta(np.arange(5), unit='s') Out[60]: TimedeltaIndex(['00:00:00', '00:00:01', '00:00:02', '00:00:03', '00:00:04'], dtype='timedelta64[ns]', freq=None) In [61]: pd.to_timedelta(np.arange(5), unit='d') Out[61]: TimedeltaIndex(['0 days', '1 days', '2 days', '3 days', '4 days'], dtype='timedelta64[ns]', freq=None)
A Series of dtype timedelta64[ns] can now be divided by another timedelta64[ns] object, or astyped to yield a float64 dtyped Series. This is frequency conversion. See the docs for the docs.
float64
In [62]: import datetime In [63]: td = pd.Series(pd.date_range('20130101', periods=4)) - pd.Series( ....: pd.date_range('20121201', periods=4)) ....: In [64]: td[2] += np.timedelta64(datetime.timedelta(minutes=5, seconds=3)) In [65]: td[3] = np.nan In [66]: td Out[66]: 0 31 days 00:00:00 1 31 days 00:00:00 2 31 days 00:05:03 3 NaT dtype: timedelta64[ns] # to days In [67]: td / np.timedelta64(1, 'D') Out[67]: 0 31.000000 1 31.000000 2 31.003507 3 NaN dtype: float64 In [68]: td.astype('timedelta64[D]') Out[68]: 0 31.0 1 31.0 2 31.0 3 NaN dtype: float64 # to seconds In [69]: td / np.timedelta64(1, 's') Out[69]: 0 2678400.0 1 2678400.0 2 2678703.0 3 NaN dtype: float64 In [70]: td.astype('timedelta64[s]') Out[70]: 0 2678400.0 1 2678400.0 2 2678703.0 3 NaN dtype: float64
Dividing or multiplying a timedelta64[ns] Series by an integer or integer Series
In [71]: td * -1 Out[71]: 0 -31 days +00:00:00 1 -31 days +00:00:00 2 -32 days +23:54:57 3 NaT dtype: timedelta64[ns] In [72]: td * pd.Series([1, 2, 3, 4]) Out[72]: 0 31 days 00:00:00 1 62 days 00:00:00 2 93 days 00:15:09 3 NaT dtype: timedelta64[ns]
Absolute DateOffset objects can act equivalently to timedeltas
DateOffset
timedeltas
In [73]: from pandas import offsets In [74]: td + offsets.Minute(5) + offsets.Milli(5) Out[74]: 0 31 days 00:05:00.005000 1 31 days 00:05:00.005000 2 31 days 00:10:03.005000 3 NaT dtype: timedelta64[ns]
Fillna is now supported for timedeltas
In [75]: td.fillna(pd.Timedelta(0)) Out[75]: 0 31 days 00:00:00 1 31 days 00:00:00 2 31 days 00:05:03 3 0 days 00:00:00 dtype: timedelta64[ns] In [76]: td.fillna(datetime.timedelta(days=1, seconds=5)) Out[76]: 0 31 days 00:00:00 1 31 days 00:00:00 2 31 days 00:05:03 3 1 days 00:00:05 dtype: timedelta64[ns]
You can do numeric reduction operations on timedeltas.
In [77]: td.mean() Out[77]: Timedelta('31 days 00:01:41') In [78]: td.quantile(.1) Out[78]: Timedelta('31 days 00:00:00')
plot(kind='kde') now accepts the optional parameters bw_method and ind, passed to scipy.stats.gaussian_kde() (for scipy >= 0.11.0) to set the bandwidth, and to gkde.evaluate() to specify the indices at which it is evaluated, respectively. See scipy docs. (GH4298)
plot(kind='kde')
bw_method
ind
DataFrame constructor now accepts a numpy masked record array (GH3478)
The new vectorized string method extract return regular expression matches more conveniently.
In [79]: pd.Series(['a1', 'b2', 'c3']).str.extract('[ab](\\d)') Out[79]: 0 0 1 1 2 2 NaN
Elements that do not match return NaN. Extracting a regular expression with more than one group returns a DataFrame with one column per group.
In [80]: pd.Series(['a1', 'b2', 'c3']).str.extract('([ab])(\\d)') Out[80]: 0 1 0 a 1 1 b 2 2 NaN NaN
Elements that do not match return a row of NaN. Thus, a Series of messy strings can be converted into a like-indexed Series or DataFrame of cleaned-up or more useful strings, without necessitating get() to access tuples or re.match objects.
get()
Named groups like
In [81]: pd.Series(['a1', 'b2', 'c3']).str.extract( ....: '(?P<letter>[ab])(?P<digit>\\d)') ....: Out[81]: letter digit 0 a 1 1 b 2 2 NaN NaN
and optional groups can also be used.
In [82]: pd.Series(['a1', 'b2', '3']).str.extract( ....: '(?P<letter>[ab])?(?P<digit>\\d)') ....: Out[82]: letter digit 0 a 1 1 b 2 2 NaN 3
read_stata now accepts Stata 13 format (GH4291)
read_stata
read_fwf now infers the column specifications from the first 100 rows of the file if the data has correctly separated and properly aligned columns using the delimiter provided to the function (GH4488).
read_fwf
support for nanosecond times as an offset
These operations require numpy >= 1.7
Period conversions in the range of seconds and below were reworked and extended up to nanoseconds. Periods in the nanosecond range are now available.
In [83]: pd.date_range('2013-01-01', periods=5, freq='5N') Out[83]: DatetimeIndex([ '2013-01-01 00:00:00', '2013-01-01 00:00:00.000000005', '2013-01-01 00:00:00.000000010', '2013-01-01 00:00:00.000000015', '2013-01-01 00:00:00.000000020'], dtype='datetime64[ns]', freq='5N')
or with frequency as offset
In [84]: pd.date_range('2013-01-01', periods=5, freq=pd.offsets.Nano(5)) Out[84]: DatetimeIndex([ '2013-01-01 00:00:00', '2013-01-01 00:00:00.000000005', '2013-01-01 00:00:00.000000010', '2013-01-01 00:00:00.000000015', '2013-01-01 00:00:00.000000020'], dtype='datetime64[ns]', freq='5N')
Timestamps can be modified in the nanosecond range
In [85]: t = pd.Timestamp('20130101 09:01:02') In [86]: t + pd.tseries.offsets.Nano(123) Out[86]: Timestamp('2013-01-01 09:01:02.000000123')
A new method, isin for DataFrames, which plays nicely with boolean indexing. The argument to isin, what we’re comparing the DataFrame to, can be a DataFrame, Series, dict, or array of values. See the docs for more.
To get the rows where any of the conditions are met:
In [87]: dfi = pd.DataFrame({'A': [1, 2, 3, 4], 'B': ['a', 'b', 'f', 'n']}) In [88]: dfi Out[88]: A B 0 1 a 1 2 b 2 3 f 3 4 n In [89]: other = pd.DataFrame({'A': [1, 3, 3, 7], 'B': ['e', 'f', 'f', 'e']}) In [90]: mask = dfi.isin(other) In [91]: mask Out[91]: A B 0 True False 1 False False 2 True True 3 False False In [92]: dfi[mask.any(1)] Out[92]: A B 0 1 a 2 3 f
Series now supports a to_frame method to convert it to a single-column DataFrame (GH5164)
to_frame
All R datasets listed here http://stat.ethz.ch/R-manual/R-devel/library/datasets/html/00Index.html can now be loaded into Pandas objects
# note that pandas.rpy was deprecated in v0.16.0 import pandas.rpy.common as com com.load_data('Titanic')
tz_localize can infer a fall daylight savings transition based on the structure of the unlocalized data (GH4230), see the docs
tz_localize
DatetimeIndex is now in the API documentation, see the docs
DatetimeIndex
json_normalize() is a new method to allow you to create a flat table from semi-structured JSON data. See the docs (GH1067)
json_normalize()
Added PySide support for the qtpandas DataFrameModel and DataFrameWidget.
Python csv parser now supports usecols (GH4335)
Frequencies gained several new offsets:
LastWeekOfMonth (GH4637)
LastWeekOfMonth
FY5253, and FY5253Quarter (GH4511)
FY5253
FY5253Quarter
DataFrame has a new interpolate method, similar to Series (GH4434, GH1892)
interpolate
In [93]: df = pd.DataFrame({'A': [1, 2.1, np.nan, 4.7, 5.6, 6.8], ....: 'B': [.25, np.nan, np.nan, 4, 12.2, 14.4]}) ....: In [94]: df.interpolate() Out[94]: A B 0 1.0 0.25 1 2.1 1.50 2 3.4 2.75 3 4.7 4.00 4 5.6 12.20 5 6.8 14.40
Additionally, the method argument to interpolate has been expanded to include 'nearest', 'zero', 'slinear', 'quadratic', 'cubic', 'barycentric', 'krogh', 'piecewise_polynomial', 'pchip', 'polynomial', 'spline' The new methods require scipy. Consult the Scipy reference guide and documentation for more information about when the various methods are appropriate. See the docs.
method
'nearest', 'zero', 'slinear', 'quadratic', 'cubic', 'barycentric', 'krogh', 'piecewise_polynomial', 'pchip', 'polynomial', 'spline'
Interpolate now also accepts a limit keyword argument. This works similar to fillna’s limit:
limit
fillna
In [95]: ser = pd.Series([1, 3, np.nan, np.nan, np.nan, 11]) In [96]: ser.interpolate(limit=2) Out[96]: 0 1.0 1 3.0 2 5.0 3 7.0 4 NaN 5 11.0 dtype: float64
Added wide_to_long panel data convenience function. See the docs.
wide_to_long
In [97]: np.random.seed(123) In [98]: df = pd.DataFrame({"A1970" : {0 : "a", 1 : "b", 2 : "c"}, ....: "A1980" : {0 : "d", 1 : "e", 2 : "f"}, ....: "B1970" : {0 : 2.5, 1 : 1.2, 2 : .7}, ....: "B1980" : {0 : 3.2, 1 : 1.3, 2 : .1}, ....: "X" : dict(zip(range(3), np.random.randn(3))) ....: }) ....: In [99]: df["id"] = df.index In [100]: df Out[100]: A1970 A1980 B1970 B1980 X id 0 a d 2.5 3.2 -1.085631 0 1 b e 1.2 1.3 0.997345 1 2 c f 0.7 0.1 0.282978 2 In [101]: pd.wide_to_long(df, ["A", "B"], i="id", j="year") Out[101]: X A B id year 0 1970 -1.085631 a 2.5 1 1970 0.997345 b 1.2 2 1970 0.282978 c 0.7 0 1980 -1.085631 d 3.2 1 1980 0.997345 e 1.3 2 1980 0.282978 f 0.1
to_csv now takes a date_format keyword argument that specifies how output datetime objects should be formatted. Datetimes encountered in the index, columns, and values will all have this formatting applied. (GH4313)
date_format
DataFrame.plot will scatter plot x versus y by passing kind='scatter' (GH2215)
DataFrame.plot
kind='scatter'
Added support for Google Analytics v3 API segment IDs that also supports v2 IDs. (GH5271)
The new eval() function implements expression evaluation using numexpr behind the scenes. This results in large speedups for complicated expressions involving large DataFrames/Series. For example,
eval()
numexpr
In [102]: nrows, ncols = 20000, 100 In [103]: df1, df2, df3, df4 = [pd.DataFrame(np.random.randn(nrows, ncols)) .....: for _ in range(4)] .....:
# eval with NumExpr backend In [104]: %timeit pd.eval('df1 + df2 + df3 + df4') 12.5 ms +- 787 us per loop (mean +- std. dev. of 7 runs, 100 loops each)
# pure Python evaluation In [105]: %timeit df1 + df2 + df3 + df4 13.1 ms +- 1.25 ms per loop (mean +- std. dev. of 7 runs, 100 loops each)
For more details, see the the docs
Similar to pandas.eval, DataFrame has a new DataFrame.eval method that evaluates an expression in the context of the DataFrame. For example,
pandas.eval
DataFrame.eval
In [106]: df = pd.DataFrame(np.random.randn(10, 2), columns=['a', 'b']) In [107]: df.eval('a + b') Out[107]: 0 -0.685204 1 1.589745 2 0.325441 3 -1.784153 4 -0.432893 5 0.171850 6 1.895919 7 3.065587 8 -0.092759 9 1.391365 dtype: float64
query() method has been added that allows you to select elements of a DataFrame using a natural query syntax nearly identical to Python syntax. For example,
query()
In [108]: n = 20 In [109]: df = pd.DataFrame(np.random.randint(n, size=(n, 3)), columns=['a', 'b', 'c']) In [110]: df.query('a < b < c') Out[110]: a b c 11 1 5 8 15 8 16 19
selects all the rows of df where a < b < c evaluates to True. For more details see the the docs.
df
a < b < c
pd.read_msgpack() and pd.to_msgpack() are now a supported method of serialization of arbitrary pandas (and python objects) in a lightweight portable binary format. See the docs
pd.read_msgpack()
pd.to_msgpack()
Since this is an EXPERIMENTAL LIBRARY, the storage format may not be stable until a future release.
df = pd.DataFrame(np.random.rand(5, 2), columns=list('AB')) df.to_msgpack('foo.msg') pd.read_msgpack('foo.msg') s = pd.Series(np.random.rand(5), index=pd.date_range('20130101', periods=5)) pd.to_msgpack('foo.msg', df, s) pd.read_msgpack('foo.msg')
You can pass iterator=True to iterator over the unpacked results
iterator=True
for o in pd.read_msgpack('foo.msg', iterator=True): print(o)
pandas.io.gbq provides a simple way to extract from, and load data into, Google’s BigQuery Data Sets by way of pandas DataFrames. BigQuery is a high performance SQL-like database service, useful for performing ad-hoc queries against extremely large datasets. See the docs
pandas.io.gbq
from pandas.io import gbq # A query to select the average monthly temperatures in the # in the year 2000 across the USA. The dataset, # publicata:samples.gsod, is available on all BigQuery accounts, # and is based on NOAA gsod data. query = """SELECT station_number as STATION, month as MONTH, AVG(mean_temp) as MEAN_TEMP FROM publicdata:samples.gsod WHERE YEAR = 2000 GROUP BY STATION, MONTH ORDER BY STATION, MONTH ASC""" # Fetch the result set for this query # Your Google BigQuery Project ID # To find this, see your dashboard: # https://console.developers.google.com/iam-admin/projects?authuser=0 projectid = 'xxxxxxxxx' df = gbq.read_gbq(query, project_id=projectid) # Use pandas to process and reshape the dataset df2 = df.pivot(index='STATION', columns='MONTH', values='MEAN_TEMP') df3 = pd.concat([df2.min(), df2.mean(), df2.max()], axis=1, keys=["Min Tem", "Mean Temp", "Max Temp"])
The resulting DataFrame is:
> df3 Min Tem Mean Temp Max Temp MONTH 1 -53.336667 39.827892 89.770968 2 -49.837500 43.685219 93.437932 3 -77.926087 48.708355 96.099998 4 -82.892858 55.070087 97.317240 5 -92.378261 61.428117 102.042856 6 -77.703334 65.858888 102.900000 7 -87.821428 68.169663 106.510714 8 -89.431999 68.614215 105.500000 9 -86.611112 63.436935 107.142856 10 -78.209677 56.880838 92.103333 11 -50.125000 48.861228 94.996428 12 -50.332258 42.286879 94.396774
To use this module, you will need a BigQuery account. See <https://cloud.google.com/products/big-query> for details.
As of 10/10/13, there is a bug in Google’s API preventing result sets from being larger than 100,000 rows. A patch is scheduled for the week of 10/14/13.
In 0.13.0 there is a major refactor primarily to subclass Series from NDFrame, which is the base class currently for DataFrame and Panel, to unify methods and behaviors. Series formerly subclassed directly from ndarray. (GH4080, GH3862, GH816)
There are two potential incompatibilities from < 0.13.0
Using certain numpy functions would previously return a Series if passed a Series as an argument. This seems only to affect np.ones_like, np.empty_like, np.diff and np.where. These now return ndarrays.
np.ones_like
np.empty_like
np.diff
np.where
ndarrays
In [111]: s = pd.Series([1, 2, 3, 4])
Numpy Usage
In [112]: np.ones_like(s) Out[112]: array([1, 1, 1, 1]) In [113]: np.diff(s) Out[113]: array([1, 1, 1]) In [114]: np.where(s > 1, s, np.nan) Out[114]: array([nan, 2., 3., 4.])
Pandonic Usage
In [115]: pd.Series(1, index=s.index) Out[115]: 0 1 1 1 2 1 3 1 dtype: int64 In [116]: s.diff() Out[116]: 0 NaN 1 1.0 2 1.0 3 1.0 dtype: float64 In [117]: s.where(s > 1) Out[117]: 0 NaN 1 2.0 2 3.0 3 4.0 dtype: float64
Passing a Series directly to a cython function expecting an ndarray type will no long work directly, you must pass Series.values, See Enhancing Performance
Series.values
Series(0.5) would previously return the scalar 0.5, instead this will return a 1-element Series
Series(0.5)
0.5
This change breaks rpy2<=2.3.8. an Issue has been opened against rpy2 and a workaround is detailed in GH5698. Thanks @JanSchulz.
rpy2<=2.3.8
Pickle compatibility is preserved for pickles created prior to 0.13. These must be unpickled with pd.read_pickle, see Pickling.
pd.read_pickle
Refactor of series.py/frame.py/panel.py to move common code to generic.py
added _setup_axes to created generic NDFrame structures
_setup_axes
moved methods
from_axes,_wrap_array,axes,ix,loc,iloc,shape,empty,swapaxes,transpose,pop
__iter__,keys,__contains__,__len__,__neg__,__invert__
convert_objects,as_blocks,as_matrix,values
__getstate__,__setstate__ (compat remains in frame/panel)
__getstate__,__setstate__
__getattr__,__setattr__
_indexed_same,reindex_like,align,where,mask
fillna,replace (Series replace is now consistent with DataFrame)
fillna,replace
filter (also added axis argument to selectively filter on a different axis)
filter
reindex,reindex_axis,take
truncate (moved to become part of NDFrame)
truncate
These are API changes which make Panel more consistent with DataFrame
swapaxes on a Panel with the same axes specified now return a copy
swapaxes
support attribute access for setting
filter supports the same API as the original DataFrame filter
Reindex called with no arguments will now return a copy of the input object
TimeSeries is now an alias for Series. the property is_time_series can be used to distinguish (if desired)
TimeSeries
is_time_series
Refactor of Sparse objects to use BlockManager
Created a new block type in internals, SparseBlock, which can hold multi-dtypes and is non-consolidatable. SparseSeries and SparseDataFrame now inherit more methods from there hierarchy (Series/DataFrame), and no longer inherit from SparseArray (which instead is the object of the SparseBlock)
SparseBlock
SparseSeries
SparseDataFrame
SparseArray
Sparse suite now supports integration with non-sparse data. Non-float sparse data is supportable (partially implemented)
Operations on sparse structures within DataFrames should preserve sparseness, merging type operations will convert to dense (and back to sparse), so might be somewhat inefficient
enable setitem on SparseSeries for boolean/integer/slices
SparsePanels implementation is unchanged (e.g. not using BlockManager, needs work)
SparsePanels
added ftypes method to Series/DataFrame, similar to dtypes, but indicates if the underlying is sparse/dense (as well as the dtype)
ftypes
dtypes
All NDFrame objects can now use __finalize__() to specify various values to propagate to new objects from an existing one (e.g. name in Series will follow more automatically now)
__finalize__()
name
Internal type checking is now done via a suite of generated classes, allowing isinstance(value, klass) without having to directly import the klass, courtesy of @jtratner
isinstance(value, klass)
Bug in Series update where the parent frame is not updating its cache based on changes (GH4080) or types (GH3217), fillna (GH3386)
Indexing with dtype conversions fixed (GH4463, GH4204)
Refactor Series.reindex to core/generic.py (GH4604, GH4618), allow method= in reindexing on a Series to work
Series.reindex
method=
Series.copy no longer accepts the order parameter and is now consistent with NDFrame copy
Series.copy
order
Refactor rename methods to core/generic.py; fixes Series.rename for (GH4605), and adds rename with the same signature for Panel
rename
Series.rename
Refactor clip methods to core/generic.py (GH4798)
clip
Refactor of _get_numeric_data/_get_bool_data to core/generic.py, allowing Series/Panel functionality
_get_numeric_data/_get_bool_data
Series (for index) / Panel (for items) now allow attribute access to its elements (GH1903)
In [118]: s = pd.Series([1, 2, 3], index=list('abc')) In [119]: s.b Out[119]: 2 In [120]: s.a = 5 In [121]: s Out[121]: a 5 b 2 c 3 dtype: int64
raising an invalid TypeError rather than ValueError when appending with a different block ordering (GH4096)
read_hdf was not respecting as passed mode (GH4504)
read_hdf
mode
appending a 0-len table will work correctly (GH4273)
to_hdf was raising when passing both arguments append and table (GH4584)
to_hdf
reading from a store with duplicate columns across dtypes would raise (GH4767)
Fixed a bug where ValueError wasn’t correctly raised when column names weren’t strings (GH4956)
A zero length series written in Fixed format not deserializing properly. (GH4708)
Fixed decoding perf issue on pyt3 (GH5441)
Validate levels in a MultiIndex before storing (GH5527)
Correctly handle data_columns with a Panel (GH5717)
data_columns
Fixed bug in tslib.tz_convert(vals, tz1, tz2): it could raise IndexError exception while trying to access trans[pos + 1] (GH4496)
The by argument now works correctly with the layout argument (GH4102, GH4014) in *.hist plotting methods
by
layout
*.hist
Fixed bug in PeriodIndex.map where using str would return the str representation of the index (GH4136)
PeriodIndex.map
str
Fixed test failure test_time_series_plot_color_with_empty_kwargs when using custom matplotlib default colors (GH4345)
test_time_series_plot_color_with_empty_kwargs
Fix running of stata IO tests. Now uses temporary files to write (GH4353)
Fixed an issue where DataFrame.sum was slower than DataFrame.mean for integer valued frames (GH4365)
DataFrame.sum
DataFrame.mean
read_html tests now work with Python 2.6 (GH4351)
Fixed bug where network testing was throwing NameError because a local variable was undefined (GH4381)
network
NameError
In to_json, raise if a passed orient would cause loss of data because of a duplicate index (GH4359)
to_json
orient
In to_json, fix date handling so milliseconds are the default timestamp as the docstring says (GH4362).
as_index is no longer ignored when doing groupby apply (GH4648, GH3417)
as_index
JSON NaT handling fixed, NaTs are now serialized to null (GH4498)
Fixed JSON handling of escapable characters in JSON object keys (GH4593)
Fixed passing keep_default_na=False when na_values=None (GH4318)
keep_default_na=False
na_values=None
Fixed bug with values raising an error on a DataFrame with duplicate columns and mixed dtypes, surfaced in (GH4377)
values
Fixed bug with duplicate columns and type conversion in read_json when orient='split' (GH4377)
read_json
orient='split'
Fixed JSON bug where locales with decimal separators other than ‘.’ threw exceptions when encoding / decoding certain values. (GH4918)
Fix .iat indexing with a PeriodIndex (GH4390)
.iat
PeriodIndex
Fixed an issue where PeriodIndex joining with self was returning a new instance rather than the same instance (GH4379); also adds a test for this for the other index types
Fixed a bug with all the dtypes being converted to object when using the CSV cparser with the usecols parameter (GH3192)
Fix an issue in merging blocks where the resulting DataFrame had partially set _ref_locs (GH4403)
Fixed an issue where hist subplots were being overwritten when they were called using the top level matplotlib API (GH4408)
Fixed a bug where calling Series.astype(str) would truncate the string (GH4405, GH4437)
Series.astype(str)
Fixed a py3 compat issue where bytes were being repr’d as tuples (GH4455)
Fixed Panel attribute naming conflict if item is named ‘a’ (GH3440)
Fixed an issue where duplicate indexes were raising when plotting (GH4486)
Fixed an issue where cumsum and cumprod didn’t work with bool dtypes (GH4170, GH4440)
Fixed Panel slicing issued in xs that was returning an incorrect dimmed object (GH4016)
xs
Fix resampling bug where custom reduce function not used if only one group (GH3849, GH4494)
Fixed Panel assignment with a transposed frame (GH3830)
Raise on set indexing with a Panel and a Panel as a value which needs alignment (GH3777)
frozenset objects now raise in the Series constructor (GH4482, GH4480)
Fixed issue with sorting a duplicate MultiIndex that has multiple dtypes (GH4516)
Fixed bug in DataFrame.set_values which was causing name attributes to be lost when expanding the index. (GH3742, GH4039)
DataFrame.set_values
Fixed issue where individual names, levels and labels could be set on MultiIndex without validation (GH3714, GH4039)
Fixed (GH3334) in pivot_table. Margins did not compute if values is the index.
Fix bug in having a rhs of np.timedelta64 or np.offsets.DateOffset when operating with datetimes (GH4532)
np.offsets.DateOffset
Fix arithmetic with series/datetimeindex and np.timedelta64 not working the same (GH4134) and buggy timedelta in NumPy 1.6 (GH4135)
Fix bug in pd.read_clipboard on windows with PY3 (GH4561); not decoding properly
tslib.get_period_field() and tslib.get_period_field_arr() now raise if code argument out of range (GH4519, GH4520)
tslib.get_period_field()
tslib.get_period_field_arr()
Fix boolean indexing on an empty series loses index names (GH4235), infer_dtype works with empty arrays.
Fix reindexing with multiple axes; if an axes match was not replacing the current axes, leading to a possible lazy frequency inference issue (GH3317)
Fixed issue where DataFrame.apply was reraising exceptions incorrectly (causing the original stack trace to be truncated).
DataFrame.apply
Fix selection with ix/loc and non_unique selectors (GH4619)
ix/loc
Fix assignment with iloc/loc involving a dtype change in an existing column (GH4312, GH5702) have internal setitem_with_indexer in core/indexing to use Block.setitem
Fixed bug where thousands operator was not handled correctly for floating point numbers in csv_import (GH4322)
Fix an issue with CacheableOffset not properly being used by many DateOffset; this prevented the DateOffset from being cached (GH4609)
Fix boolean comparison with a DataFrame on the lhs, and a list/tuple on the rhs (GH4576)
Fix error/dtype conversion with setitem of None on Series/DataFrame (GH4667)
None
Series/DataFrame
Fix decoding based on a passed in non-default encoding in pd.read_stata (GH4626)
pd.read_stata
Fix DataFrame.from_records with a plain-vanilla ndarray. (GH4727)
DataFrame.from_records
Fix some inconsistencies with Index.rename and MultiIndex.rename, etc. (GH4718, GH4628)
Index.rename
MultiIndex.rename
Bug in using iloc/loc with a cross-sectional and duplicate indices (GH4726)
iloc/loc
Bug with using QUOTE_NONE with to_csv causing Exception. (GH4328)
QUOTE_NONE
Exception
Bug with Series indexing not raising an error when the right-hand-side has an incorrect length (GH2702)
Bug in MultiIndexing with a partial string selection as one part of a MultIndex (GH4758)
Bug with reindexing on the index with a non-unique index will now raise ValueError (GH4746)
Bug in setting with loc/ix a single indexer with a MultiIndex axis and a NumPy array, related to (GH3777)
loc/ix
Bug in concatenation with duplicate columns across dtypes not merging with axis=0 (GH4771, GH4975)
Bug in iloc with a slice index failing (GH4771)
Incorrect error message with no colspecs or width in read_fwf. (GH4774)
Fix bugs in indexing in a Series with a duplicate index (GH4548, GH4550)
Fixed bug with reading compressed files with read_fwf in Python 3. (GH3963)
Fixed an issue with a duplicate index and assignment with a dtype change (GH4686)
Fixed bug with reading compressed files in as bytes rather than str in Python 3. Simplifies bytes-producing file-handling in Python 3 (GH3963, GH4785).
bytes
Fixed an issue related to ticklocs/ticklabels with log scale bar plots across different versions of matplotlib (GH4789)
Suppressed DeprecationWarning associated with internal calls issued by repr() (GH4391)
Fixed an issue with a duplicate index and duplicate selector with .loc (GH4825)
.loc
Fixed an issue with DataFrame.sort_index where, when sorting by a single column and passing a list for ascending, the argument for ascending was being interpreted as True (GH4839, GH4846)
DataFrame.sort_index
ascending
Fixed Panel.tshift not working. Added freq support to Panel.shift (GH4853)
Panel.tshift
Panel.shift
Fix an issue in TextFileReader w/ Python engine (i.e. PythonParser) with thousands != “,” (GH4596)
Bug in getitem with a duplicate index when using where (GH4879)
Fix Type inference code coerces float column into datetime (GH4601)
Fixed _ensure_numeric does not check for complex numbers (GH4902)
_ensure_numeric
Fixed a bug in Series.hist where two figures were being created when the by argument was passed (GH4112, GH4113).
Series.hist
Fixed a bug in convert_objects for > 2 ndims (GH4937)
convert_objects
Fixed a bug in DataFrame/Panel cache insertion and subsequent indexing (GH4939, GH5424)
Fixed string methods for FrozenNDArray and FrozenList (GH4929)
FrozenNDArray
FrozenList
Fixed a bug with setting invalid or out-of-range values in indexing enlargement scenarios (GH4940)
Tests for fillna on empty Series (GH4346), thanks @immerrr
Fixed copy() to shallow copy axes/indices as well and thereby keep separate metadata. (GH4202, GH4830)
copy()
Fixed skiprows option in Python parser for read_csv (GH4382)
Fixed bug preventing cut from working with np.inf levels without explicitly passing labels (GH3415)
cut
np.inf
Fixed wrong check for overlapping in DatetimeIndex.union (GH4564)
DatetimeIndex.union
Fixed conflict between thousands separator and date parser in csv_parser (GH4678)
Fix appending when dtypes are not the same (error showing mixing float/np.datetime64) (GH4993)
Fix repr for DateOffset. No longer show duplicate entries in kwds. Removed unused offset fields. (GH4638)
Fixed wrong index name during read_csv if using usecols. Applies to c parser only. (GH4201)
Timestamp objects can now appear in the left hand side of a comparison operation with a Series or DataFrame object (GH4982).
Timestamp
Fix a bug when indexing with np.nan via iloc/loc (GH5016)
np.nan
Fixed a bug where low memory c parser could create different types in different chunks of the same file. Now coerces to numerical type or raises warning. (GH3866)
Fix a bug where reshaping a Series to its own shape raised TypeError (GH4554) and other reshaping issues.
Bug in setting with ix/loc and a mixed int/string index (GH4544)
Make sure series-series boolean comparisons are label based (GH4947)
Bug in multi-level indexing with a Timestamp partial indexer (GH4294)
Tests/fix for MultiIndex construction of an all-nan frame (GH4078)
Fixed a bug where read_html() wasn’t correctly inferring values of tables with commas (GH5029)
read_html()
Fixed a bug where read_html() wasn’t providing a stable ordering of returned tables (GH4770, GH5029).
Fixed a bug where read_html() was incorrectly parsing when passed index_col=0 (GH5066).
index_col=0
Fixed a bug where read_html() was incorrectly inferring the type of headers (GH5048).
Fixed a bug where DatetimeIndex joins with PeriodIndex caused a stack overflow (GH3899).
Fixed a bug where groupby objects didn’t allow plots (GH5102).
groupby
Fixed a bug where groupby objects weren’t tab-completing column names (GH5102).
Fixed a bug where groupby.plot() and friends were duplicating figures multiple times (GH5102).
groupby.plot()
Provide automatic conversion of object dtypes on fillna, related (GH5103)
object
Fixed a bug where default options were being overwritten in the option parser cleaning (GH5121).
Treat a list/ndarray identically for iloc indexing with list-like (GH5006)
Fix MultiIndex.get_level_values() with missing values (GH5074)
MultiIndex.get_level_values()
Fix bound checking for Timestamp() with datetime64 input (GH4065)
Fix a bug where TestReadHtml wasn’t calling the correct read_html() function (GH5150).
TestReadHtml
Fix a bug with NDFrame.replace() which made replacement appear as though it was (incorrectly) using regular expressions (GH5143).
NDFrame.replace()
Fix better error message for to_datetime (GH4928)
Made sure different locales are tested on travis-ci (GH4918). Also adds a couple of utilities for getting locales and setting locales with a context manager.
Fixed segfault on isnull(MultiIndex) (now raises an error instead) (GH5123, GH5125)
isnull(MultiIndex)
Allow duplicate indices when performing operations that align (GH5185, GH5639)
Compound dtypes in a constructor raise NotImplementedError (GH5191)
NotImplementedError
Bug in comparing duplicate frames (GH4421) related
Bug in describe on duplicate frames
Bug in to_datetime with a format and coerce=True not raising (GH5195)
to_datetime
coerce=True
Bug in loc setting with multiple indexers and a rhs of a Series that needs broadcasting (GH5206)
loc
Fixed bug where inplace setting of levels or labels on MultiIndex would not clear cached values property and therefore return wrong values. (GH5215)
Fixed bug where filtering a grouped DataFrame or Series did not maintain the original ordering (GH4621).
Fixed Period with a business date freq to always roll-forward if on a non-business date. (GH5203)
Period
Fixed bug in Excel writers where frames with duplicate column names weren’t written correctly. (GH5235)
Fixed issue with drop and a non-unique index on Series (GH5248)
drop
Fixed segfault in C parser caused by passing more names than columns in the file. (GH5156)
Fix Series.isin with date/time-like dtypes (GH5021)
Series.isin
C and Python Parser can now handle the more common MultiIndex column format which doesn’t have a row for index names (GH4702)
Bug when trying to use an out-of-bounds date as an object dtype (GH5312)
Bug when trying to display an embedded PandasObject (GH5324)
Allows operating of Timestamps to return a datetime if the result is out-of-bounds related (GH5312)
Fix return value/type signature of initObjToJSON() to be compatible with numpy’s import_array() (GH5334, GH5326)
initObjToJSON()
import_array()
Bug when renaming then set_index on a DataFrame (GH5344)
Test suite no longer leaves around temporary files when testing graphics. (GH5347) (thanks for catching this @yarikoptic!)
Fixed html tests on win32. (GH4580)
Make sure that head/tail are iloc based, (GH5370)
head/tail
Fixed bug for PeriodIndex string representation if there are 1 or 2 elements. (GH5372)
The GroupBy methods transform and filter can be used on Series and DataFrames that have repeated (non-unique) indices. (GH4620)
transform
Fix empty series not printing name in repr (GH4651)
Make tests create temp files in temp directory by default. (GH5419)
pd.to_timedelta of a scalar returns a scalar (GH5410)
pd.to_timedelta
pd.to_timedelta accepts NaN and NaT, returning NaT instead of raising (GH5437)
NaT
performance improvements in isnull on larger size pandas objects
isnull
Fixed various setitem with 1d ndarray that does not have a matching length to the indexer (GH5508)
Bug in getitem with a MultiIndex and iloc (GH5528)
Bug in delitem on a Series (GH5542)
Bug fix in apply when using custom function and objects are not mutated (GH5545)
Bug in selecting from a non-unique index with loc (GH5553)
Bug in groupby returning non-consistent types when user function returns a None, (GH5592)
Work around regression in numpy 1.7.0 which erroneously raises IndexError from ndarray.item (GH5666)
ndarray.item
Bug in repeated indexing of object with resultant non-unique index (GH5678)
Bug in fillna with Series and a passed series/dict (GH5703)
Bug in groupby transform with a datetime-like grouper (GH5712)
Bug in MultiIndex selection in PY3 when using certain keys (GH5725)
Row-wise concat of differing dtypes failing in certain cases (GH5754)
A total of 77 people contributed patches to this release. People with a “+” by their names contributed a patch for the first time.
Agustín Herranz +
Alex Gaudio +
Alex Rothberg +
Andreas Klostermann +
Andreas Würl +
Andy Hayden
Ben Alex +
Benedikt Sauer +
Brad Buran
Caleb Epstein +
Chang She
Christopher Whelan
DSM +
Dale Jung +
Dan Birken
David Rasch +
Dieter Vandenbussche
Gabi Davar +
Garrett Drapala
Goyo +
Greg Reda +
Ivan Smirnov +
Jack Kelly +
Jacob Schaer +
Jan Schulz +
Jeff Tratner
Jeffrey Tratner
John McNamara +
John W. O’Brien +
Joris Van den Bossche
Justin Bozonier +
Kelsey Jordahl
Kevin Stone
Kieran O’Mahony
Kyle Hausmann +
Kyle Kelley +
Kyle Meyer
Mike Kelly
Mortada Mehyar +
Nick Foti +
Olivier Harris +
Ondřej Čertík +
PKEuS
Phillip Cloud
Pierre Haessig +
Richard T. Guy +
Roman Pekar +
Roy Hyunjin Han
Skipper Seabold
Sten +
Thomas A Caswell +
Thomas Kluyver
Tiago Requeijo +
TomAugspurger
Trent Hauck
Valentin Haenel +
Viktor Kerkez +
Vincent Arel-Bundock
Wes McKinney
Wes Turner +
Weston Renoud +
Yaroslav Halchenko
Zach Dwiel +
chapman siu +
chappers +
d10genes +
danielballan
daydreamt +
engstrom +
jreback
monicaBee +
prossahl +
rockg +
unutbu +
westurner +
y-p
zach powers