This is a minor bug-fix release from 0.18.0 and includes a large number of bug fixes along with several new features, enhancements, and performance improvements. We recommend that all users upgrade to this version.
Highlights include:
.groupby(...) has been enhanced to provide convenient syntax when working with .rolling(..), .expanding(..) and .resample(..) per group, see here
.groupby(...)
.rolling(..)
.expanding(..)
.resample(..)
pd.to_datetime() has gained the ability to assemble dates from a DataFrame, see here
pd.to_datetime()
DataFrame
Method chaining improvements, see here.
Custom business hour offset, see here.
Many bug fixes in the handling of sparse, see here
sparse
Expanded the Tutorials section with a feature on modern pandas, courtesy of @TomAugsburger. (GH13045).
What’s new in v0.18.1
New features
Custom business hour
.groupby(..) syntax with window and resample operations
.groupby(..)
Method chaining improvements
.where() and .mask()
.where()
.mask()
.loc[], .iloc[], .ix[]
.loc[]
.iloc[]
.ix[]
[] indexing
[]
Partial string indexing on DateTimeIndex when part of a MultiIndex
DateTimeIndex
MultiIndex
Assembling datetimes
Other enhancements
Sparse changes
API changes
.groupby(..).nth() changes
.groupby(..).nth()
numpy function compatibility
Using .apply on groupby resampling
.apply
Changes in read_csv exceptions
read_csv
to_datetime error changes
to_datetime
Other API changes
Deprecations
Performance improvements
Bug fixes
Contributors
The CustomBusinessHour is a mixture of BusinessHour and CustomBusinessDay which allows you to specify arbitrary holidays. For details, see Custom Business Hour (GH11514)
CustomBusinessHour
BusinessHour
CustomBusinessDay
In [1]: from pandas.tseries.offsets import CustomBusinessHour In [2]: from pandas.tseries.holiday import USFederalHolidayCalendar In [3]: bhour_us = CustomBusinessHour(calendar=USFederalHolidayCalendar())
Friday before MLK Day
In [4]: import datetime In [5]: dt = datetime.datetime(2014, 1, 17, 15) In [6]: dt + bhour_us Out[6]: Timestamp('2014-01-17 16:00:00')
Tuesday after MLK Day (Monday is skipped because it’s a holiday)
In [7]: dt + bhour_us * 2 Out[7]: Timestamp('2014-01-20 09:00:00')
.groupby(...) has been enhanced to provide convenient syntax when working with .rolling(..), .expanding(..) and .resample(..) per group, see (GH12486, GH12738).
You can now use .rolling(..) and .expanding(..) as methods on groupbys. These return another deferred object (similar to what .rolling() and .expanding() do on ungrouped pandas objects). You can then operate on these RollingGroupby objects in a similar manner.
.rolling()
.expanding()
RollingGroupby
Previously you would have to do this to get a rolling window mean per-group:
In [8]: df = pd.DataFrame({'A': [1] * 20 + [2] * 12 + [3] * 8, ...: 'B': np.arange(40)}) ...: In [9]: df Out[9]: A B 0 1 0 1 1 1 2 1 2 3 1 3 4 1 4 .. .. .. 35 3 35 36 3 36 37 3 37 38 3 38 39 3 39 [40 rows x 2 columns]
In [10]: df.groupby('A').apply(lambda x: x.rolling(4).B.mean()) Out[10]: A 1 0 NaN 1 NaN 2 NaN 3 1.5 4 2.5 ... 3 35 33.5 36 34.5 37 35.5 38 36.5 39 37.5 Name: B, Length: 40, dtype: float64
Now you can do:
In [11]: df.groupby('A').rolling(4).B.mean() Out[11]: A 1 0 NaN 1 NaN 2 NaN 3 1.5 4 2.5 ... 3 35 33.5 36 34.5 37 35.5 38 36.5 39 37.5 Name: B, Length: 40, dtype: float64
For .resample(..) type of operations, previously you would have to:
In [12]: df = pd.DataFrame({'date': pd.date_range(start='2016-01-01', ....: periods=4, ....: freq='W'), ....: 'group': [1, 1, 2, 2], ....: 'val': [5, 6, 7, 8]}).set_index('date') ....: In [13]: df Out[13]: group val date 2016-01-03 1 5 2016-01-10 1 6 2016-01-17 2 7 2016-01-24 2 8 [4 rows x 2 columns]
In [14]: df.groupby('group').apply(lambda x: x.resample('1D').ffill()) Out[14]: group val group date 1 2016-01-03 1 5 2016-01-04 1 5 2016-01-05 1 5 2016-01-06 1 5 2016-01-07 1 5 ... ... ... 2 2016-01-20 2 7 2016-01-21 2 7 2016-01-22 2 7 2016-01-23 2 7 2016-01-24 2 8 [16 rows x 2 columns]
In [15]: df.groupby('group').resample('1D').ffill() Out[15]: group val group date 1 2016-01-03 1 5 2016-01-04 1 5 2016-01-05 1 5 2016-01-06 1 5 2016-01-07 1 5 ... ... ... 2 2016-01-20 2 7 2016-01-21 2 7 2016-01-22 2 7 2016-01-23 2 7 2016-01-24 2 8 [16 rows x 2 columns]
The following methods / indexers now accept a callable. It is intended to make these more useful in method chains, see the documentation. (GH11485, GH12533)
callable
.loc[], iloc[] and .ix[]
iloc[]
These can accept a callable for the condition and other arguments.
other
In [16]: df = pd.DataFrame({'A': [1, 2, 3], ....: 'B': [4, 5, 6], ....: 'C': [7, 8, 9]}) ....: In [17]: df.where(lambda x: x > 4, lambda x: x + 10) Out[17]: A B C 0 11 14 7 1 12 5 8 2 13 6 9 [3 rows x 3 columns]
These can accept a callable, and a tuple of callable as a slicer. The callable can return a valid boolean indexer or anything which is valid for these indexer’s input.
# callable returns bool indexer In [18]: df.loc[lambda x: x.A >= 2, lambda x: x.sum() > 10] Out[18]: B C 1 5 8 2 6 9 [2 rows x 2 columns] # callable returns list of labels In [19]: df.loc[lambda x: [1, 2], lambda x: ['A', 'B']] Out[19]: A B 1 2 5 2 3 6 [2 rows x 2 columns]
Finally, you can use a callable in [] indexing of Series, DataFrame and Panel. The callable must return a valid input for [] indexing depending on its class and index type.
In [20]: df[lambda x: 'A'] Out[20]: 0 1 1 2 2 3 Name: A, Length: 3, dtype: int64
Using these methods / indexers, you can chain data selection operations without using temporary variable.
In [21]: bb = pd.read_csv('data/baseball.csv', index_col='id') In [22]: (bb.groupby(['year', 'team']) ....: .sum() ....: .loc[lambda df: df.r > 100]) ....: Out[22]: stint g ab r h X2b X3b hr rbi sb cs bb so ibb hbp sh sf gidp year team 2007 CIN 6 379 745 101 203 35 2 36 125.0 10.0 1.0 105 127.0 14.0 1.0 1.0 15.0 18.0 DET 5 301 1062 162 283 54 4 37 144.0 24.0 7.0 97 176.0 3.0 10.0 4.0 8.0 28.0 HOU 4 311 926 109 218 47 6 14 77.0 10.0 4.0 60 212.0 3.0 9.0 16.0 6.0 17.0 LAN 11 413 1021 153 293 61 3 36 154.0 7.0 5.0 114 141.0 8.0 9.0 3.0 8.0 29.0 NYN 13 622 1854 240 509 101 3 61 243.0 22.0 4.0 174 310.0 24.0 23.0 18.0 15.0 48.0 SFN 5 482 1305 198 337 67 6 40 171.0 26.0 7.0 235 188.0 51.0 8.0 16.0 6.0 41.0 TEX 2 198 729 115 200 40 4 28 115.0 21.0 4.0 73 140.0 4.0 5.0 2.0 8.0 16.0 TOR 4 459 1408 187 378 96 2 58 223.0 4.0 2.0 190 265.0 16.0 12.0 4.0 16.0 38.0 [8 rows x 18 columns]
Partial string indexing now matches on DateTimeIndex when part of a MultiIndex (GH10331)
In [23]: dft2 = pd.DataFrame( ....: np.random.randn(20, 1), ....: columns=['A'], ....: index=pd.MultiIndex.from_product([pd.date_range('20130101', ....: periods=10, ....: freq='12H'), ....: ['a', 'b']])) ....: In [24]: dft2 Out[24]: A 2013-01-01 00:00:00 a 0.469112 b -0.282863 2013-01-01 12:00:00 a -1.509059 b -1.135632 2013-01-02 00:00:00 a 1.212112 ... ... 2013-01-04 12:00:00 b 0.271860 2013-01-05 00:00:00 a -0.424972 b 0.567020 2013-01-05 12:00:00 a 0.276232 b -1.087401 [20 rows x 1 columns] In [25]: dft2.loc['2013-01-05'] Out[25]: A 2013-01-05 00:00:00 a -0.424972 b 0.567020 2013-01-05 12:00:00 a 0.276232 b -1.087401 [4 rows x 1 columns]
On other levels
In [26]: idx = pd.IndexSlice In [27]: dft2 = dft2.swaplevel(0, 1).sort_index() In [28]: dft2 Out[28]: A a 2013-01-01 00:00:00 0.469112 2013-01-01 12:00:00 -1.509059 2013-01-02 00:00:00 1.212112 2013-01-02 12:00:00 0.119209 2013-01-03 00:00:00 -0.861849 ... ... b 2013-01-03 12:00:00 1.071804 2013-01-04 00:00:00 -0.706771 2013-01-04 12:00:00 0.271860 2013-01-05 00:00:00 0.567020 2013-01-05 12:00:00 -1.087401 [20 rows x 1 columns] In [29]: dft2.loc[idx[:, '2013-01-05'], :] Out[29]: A a 2013-01-05 00:00:00 -0.424972 2013-01-05 12:00:00 0.276232 b 2013-01-05 00:00:00 0.567020 2013-01-05 12:00:00 -1.087401 [4 rows x 1 columns]
pd.to_datetime() has gained the ability to assemble datetimes from a passed in DataFrame or a dict. (GH8158).
In [30]: df = pd.DataFrame({'year': [2015, 2016], ....: 'month': [2, 3], ....: 'day': [4, 5], ....: 'hour': [2, 3]}) ....: In [31]: df Out[31]: year month day hour 0 2015 2 4 2 1 2016 3 5 3 [2 rows x 4 columns]
Assembling using the passed frame.
In [32]: pd.to_datetime(df) Out[32]: 0 2015-02-04 02:00:00 1 2016-03-05 03:00:00 Length: 2, dtype: datetime64[ns]
You can pass only the columns that you need to assemble.
In [33]: pd.to_datetime(df[['year', 'month', 'day']]) Out[33]: 0 2015-02-04 1 2016-03-05 Length: 2, dtype: datetime64[ns]
pd.read_csv() now supports delim_whitespace=True for the Python engine (GH12958)
pd.read_csv()
delim_whitespace=True
pd.read_csv() now supports opening ZIP files that contains a single CSV, via extension inference or explicit compression='zip' (GH12175)
compression='zip'
pd.read_csv() now supports opening files using xz compression, via extension inference or explicit compression='xz' is specified; xz compressions is also supported by DataFrame.to_csv in the same way (GH11852)
compression='xz'
xz
DataFrame.to_csv
pd.read_msgpack() now always gives writeable ndarrays even when compression is used (GH12359).
pd.read_msgpack()
pd.read_msgpack() now supports serializing and de-serializing categoricals with msgpack (GH12573)
.to_json() now supports NDFrames that contain categorical and sparse data (GH10778)
.to_json()
NDFrames
interpolate() now supports method='akima' (GH7588).
interpolate()
method='akima'
pd.read_excel() now accepts path objects (e.g. pathlib.Path, py.path.local) for the file path, in line with other read_* functions (GH12655)
pd.read_excel()
pathlib.Path
py.path.local
read_*
Added .weekday_name property as a component to DatetimeIndex and the .dt accessor. (GH11128)
.weekday_name
DatetimeIndex
.dt
Index.take now handles allow_fill and fill_value consistently (GH12631)
Index.take
allow_fill
fill_value
In [34]: idx = pd.Index([1., 2., 3., 4.], dtype='float') # default, allow_fill=True, fill_value=None In [35]: idx.take([2, -1]) Out[35]: Float64Index([3.0, 4.0], dtype='float64') In [36]: idx.take([2, -1], fill_value=True) Out[36]: Float64Index([3.0, nan], dtype='float64')
Index now supports .str.get_dummies() which returns MultiIndex, see Creating Indicator Variables (GH10008, GH10103)
Index
.str.get_dummies()
In [37]: idx = pd.Index(['a|b', 'a|c', 'b|c']) In [38]: idx.str.get_dummies('|') Out[38]: MultiIndex([(1, 1, 0), (1, 0, 1), (0, 1, 1)], names=['a', 'b', 'c'])
pd.crosstab() has gained a normalize argument for normalizing frequency tables (GH12569). Examples in the updated docs here.
pd.crosstab()
normalize
.resample(..).interpolate() is now supported (GH12925)
.resample(..).interpolate()
.isin() now accepts passed sets (GH12988)
.isin()
sets
These changes conform sparse handling to return the correct types and work to make a smoother experience with indexing.
SparseArray.take now returns a scalar for scalar input, SparseArray for others. Furthermore, it handles a negative indexer with the same rule as Index (GH10560, GH12796)
SparseArray.take
SparseArray
s = pd.SparseArray([np.nan, np.nan, 1, 2, 3, np.nan, 4, 5, np.nan, 6]) s.take(0) s.take([1, 2, 3])
Bug in SparseSeries[] indexing with Ellipsis raises KeyError (GH9467)
SparseSeries[]
Ellipsis
KeyError
Bug in SparseArray[] indexing with tuples are not handled properly (GH12966)
SparseArray[]
Bug in SparseSeries.loc[] with list-like input raises TypeError (GH10560)
SparseSeries.loc[]
TypeError
Bug in SparseSeries.iloc[] with scalar input may raise IndexError (GH10560)
SparseSeries.iloc[]
IndexError
Bug in SparseSeries.loc[], .iloc[] with slice returns SparseArray, rather than SparseSeries (GH10560)
slice
SparseSeries
Bug in SparseDataFrame.loc[], .iloc[] may results in dense Series, rather than SparseSeries (GH12787)
SparseDataFrame.loc[]
Series
Bug in SparseArray addition ignores fill_value of right hand side (GH12910)
Bug in SparseArray mod raises AttributeError (GH12910)
AttributeError
Bug in SparseArray pow calculates 1 ** np.nan as np.nan which must be 1 (GH12910)
1 ** np.nan
np.nan
Bug in SparseArray comparison output may incorrect result or raise ValueError (GH12971)
ValueError
Bug in SparseSeries.__repr__ raises TypeError when it is longer than max_rows (GH10560)
SparseSeries.__repr__
max_rows
Bug in SparseSeries.shape ignores fill_value (GH10452)
SparseSeries.shape
Bug in SparseSeries and SparseArray may have different dtype from its dense values (GH12908)
dtype
Bug in SparseSeries.reindex incorrectly handle fill_value (GH12797)
SparseSeries.reindex
Bug in SparseArray.to_frame() results in DataFrame, rather than SparseDataFrame (GH9850)
SparseArray.to_frame()
SparseDataFrame
Bug in SparseSeries.value_counts() does not count fill_value (GH6749)
SparseSeries.value_counts()
Bug in SparseArray.to_dense() does not preserve dtype (GH10648)
SparseArray.to_dense()
Bug in SparseArray.to_dense() incorrectly handle fill_value (GH12797)
Bug in pd.concat() of SparseSeries results in dense (GH10536)
pd.concat()
Bug in pd.concat() of SparseDataFrame incorrectly handle fill_value (GH9765)
Bug in pd.concat() of SparseDataFrame may raise AttributeError (GH12174)
Bug in SparseArray.shift() may raise NameError or TypeError (GH12908)
SparseArray.shift()
NameError
The index in .groupby(..).nth() output is now more consistent when the as_index argument is passed (GH11039):
as_index
In [39]: df = pd.DataFrame({'A': ['a', 'b', 'a'], ....: 'B': [1, 2, 3]}) ....: In [40]: df Out[40]: A B 0 a 1 1 b 2 2 a 3 [3 rows x 2 columns]
Previous behavior:
In [3]: df.groupby('A', as_index=True)['B'].nth(0) Out[3]: 0 1 1 2 Name: B, dtype: int64 In [4]: df.groupby('A', as_index=False)['B'].nth(0) Out[4]: 0 1 1 2 Name: B, dtype: int64
New behavior:
In [41]: df.groupby('A', as_index=True)['B'].nth(0) Out[41]: A a 1 b 2 Name: B, Length: 2, dtype: int64 In [42]: df.groupby('A', as_index=False)['B'].nth(0) Out[42]: 0 1 1 2 Name: B, Length: 2, dtype: int64
Furthermore, previously, a .groupby would always sort, regardless if sort=False was passed with .nth().
.groupby
sort=False
.nth()
In [43]: np.random.seed(1234) In [44]: df = pd.DataFrame(np.random.randn(100, 2), columns=['a', 'b']) In [45]: df['c'] = np.random.randint(0, 4, 100)
In [4]: df.groupby('c', sort=True).nth(1) Out[4]: a b c 0 -0.334077 0.002118 1 0.036142 -2.074978 2 -0.720589 0.887163 3 0.859588 -0.636524 In [5]: df.groupby('c', sort=False).nth(1) Out[5]: a b c 0 -0.334077 0.002118 1 0.036142 -2.074978 2 -0.720589 0.887163 3 0.859588 -0.636524
In [46]: df.groupby('c', sort=True).nth(1) Out[46]: a b c 0 -0.334077 0.002118 1 0.036142 -2.074978 2 -0.720589 0.887163 3 0.859588 -0.636524 [4 rows x 2 columns] In [47]: df.groupby('c', sort=False).nth(1) Out[47]: a b c 2 -0.720589 0.887163 3 0.859588 -0.636524 0 -0.334077 0.002118 1 0.036142 -2.074978 [4 rows x 2 columns]
Compatibility between pandas array-like methods (e.g. sum and take) and their numpy counterparts has been greatly increased by augmenting the signatures of the pandas methods so as to accept arguments that can be passed in from numpy, even if they are not necessarily used in the pandas implementation (GH12644, GH12638, GH12687)
sum
take
numpy
pandas
.searchsorted() for Index and TimedeltaIndex now accept a sorter argument to maintain compatibility with numpy’s searchsorted function (GH12238)
.searchsorted()
TimedeltaIndex
sorter
searchsorted
Bug in numpy compatibility of np.round() on a Series (GH12600)
np.round()
An example of this signature augmentation is illustrated below:
sp = pd.SparseDataFrame([1, 2, 3]) sp
Previous behaviour:
In [2]: np.cumsum(sp, axis=0) ... TypeError: cumsum() takes at most 2 arguments (4 given)
New behaviour:
np.cumsum(sp, axis=0)
Using apply on resampling groupby operations (using a pd.TimeGrouper) now has the same output types as similar apply calls on other groupby operations. (GH11742).
apply
pd.TimeGrouper
In [48]: df = pd.DataFrame({'date': pd.to_datetime(['10/10/2000', '11/10/2000']), ....: 'value': [10, 13]}) ....: In [49]: df Out[49]: date value 0 2000-10-10 10 1 2000-11-10 13 [2 rows x 2 columns]
In [1]: df.groupby(pd.TimeGrouper(key='date', ...: freq='M')).apply(lambda x: x.value.sum()) Out[1]: ... TypeError: cannot concatenate a non-NDFrame object # Output is a Series In [2]: df.groupby(pd.TimeGrouper(key='date', ...: freq='M')).apply(lambda x: x[['value']].sum()) Out[2]: date 2000-10-31 value 10 2000-11-30 value 13 dtype: int64
# Output is a Series In [55]: df.groupby(pd.TimeGrouper(key='date', ...: freq='M')).apply(lambda x: x.value.sum()) Out[55]: date 2000-10-31 10 2000-11-30 13 Freq: M, dtype: int64 # Output is a DataFrame In [56]: df.groupby(pd.TimeGrouper(key='date', ...: freq='M')).apply(lambda x: x[['value']].sum()) Out[56]: value date 2000-10-31 10 2000-11-30 13
In order to standardize the read_csv API for both the c and python engines, both will now raise an EmptyDataError, a subclass of ValueError, in response to empty columns or header (GH12493, GH12506)
c
python
EmptyDataError
In [1]: import io In [2]: df = pd.read_csv(io.StringIO(''), engine='c') ... ValueError: No columns to parse from file In [3]: df = pd.read_csv(io.StringIO(''), engine='python') ... StopIteration
In [1]: df = pd.read_csv(io.StringIO(''), engine='c') ... pandas.io.common.EmptyDataError: No columns to parse from file In [2]: df = pd.read_csv(io.StringIO(''), engine='python') ... pandas.io.common.EmptyDataError: No columns to parse from file
In addition to this error change, several others have been made as well:
CParserError now sub-classes ValueError instead of just a Exception (GH12551)
CParserError
Exception
A CParserError is now raised instead of a generic Exception in read_csv when the c engine cannot parse a column (GH12506)
A ValueError is now raised instead of a generic Exception in read_csv when the c engine encounters a NaN value in an integer column (GH12506)
NaN
A ValueError is now raised instead of a generic Exception in read_csv when true_values is specified, and the c engine encounters an element in a column containing unencodable bytes (GH12506)
true_values
pandas.parser.OverflowError exception has been removed and has been replaced with Python’s built-in OverflowError exception (GH12506)
pandas.parser.OverflowError
OverflowError
pd.read_csv() no longer allows a combination of strings and integers for the usecols parameter (GH12678)
usecols
Bugs in pd.to_datetime() when passing a unit with convertible entries and errors='coerce' or non-convertible with errors='ignore'. Furthermore, an OutOfBoundsDateime exception will be raised when an out-of-range value is encountered for that unit when errors='raise'. (GH11758, GH13052, GH13059)
unit
errors='coerce'
errors='ignore'
OutOfBoundsDateime
errors='raise'
In [27]: pd.to_datetime(1420043460, unit='s', errors='coerce') Out[27]: NaT In [28]: pd.to_datetime(11111111, unit='D', errors='ignore') OverflowError: Python int too large to convert to C long In [29]: pd.to_datetime(11111111, unit='D', errors='raise') OverflowError: Python int too large to convert to C long
In [2]: pd.to_datetime(1420043460, unit='s', errors='coerce') Out[2]: Timestamp('2014-12-31 16:31:00') In [3]: pd.to_datetime(11111111, unit='D', errors='ignore') Out[3]: 11111111 In [4]: pd.to_datetime(11111111, unit='D', errors='raise') OutOfBoundsDatetime: cannot convert input with unit 'D'
.swaplevel() for Series, DataFrame, Panel, and MultiIndex now features defaults for its first two parameters i and j that swap the two innermost levels of the index. (GH12934)
.swaplevel()
Panel
i
j
Period and PeriodIndex now raises IncompatibleFrequency error which inherits ValueError rather than raw ValueError (GH12615)
Period
PeriodIndex
IncompatibleFrequency
Series.apply for category dtype now applies the passed function to each of the .categories (and not the .codes), and returns a category dtype if possible (GH12473)
Series.apply
.categories
.codes
category
read_csv will now raise a TypeError if parse_dates is neither a boolean, list, or dictionary (matches the doc-string) (GH5636)
parse_dates
The default for .query()/.eval() is now engine=None, which will use numexpr if it’s installed; otherwise it will fallback to the python engine. This mimics the pre-0.18.1 behavior if numexpr is installed (and which, previously, if numexpr was not installed, .query()/.eval() would raise). (GH12749)
.query()/.eval()
engine=None
numexpr
pd.show_versions() now includes pandas_datareader version (GH12740)
pd.show_versions()
pandas_datareader
Provide a proper __name__ and __qualname__ attributes for generic functions (GH12021)
__name__
__qualname__
pd.concat(ignore_index=True) now uses RangeIndex as default (GH12695)
pd.concat(ignore_index=True)
RangeIndex
pd.merge() and DataFrame.join() will show a UserWarning when merging/joining a single- with a multi-leveled dataframe (GH9455, GH12219)
pd.merge()
DataFrame.join()
UserWarning
Compat with scipy > 0.17 for deprecated piecewise_polynomial interpolation method; support for the replacement from_derivatives method (GH12887)
scipy
piecewise_polynomial
from_derivatives
The method name Index.sym_diff() is deprecated and can be replaced by Index.symmetric_difference() (GH12591)
Index.sym_diff()
Index.symmetric_difference()
The method name Categorical.sort() is deprecated in favor of Categorical.sort_values() (GH12882)
Categorical.sort()
Categorical.sort_values()
Improved speed of SAS reader (GH12656, GH12961)
Performance improvements in .groupby(..).cumcount() (GH11039)
.groupby(..).cumcount()
Improved memory usage in pd.read_csv() when using skiprows=an_integer (GH13005)
skiprows=an_integer
Improved performance of DataFrame.to_sql when checking case sensitivity for tables. Now only checks if table has been created correctly when table name is not lower case. (GH12876)
DataFrame.to_sql
Improved performance of Period construction and time series plotting (GH12903, GH11831).
Improved performance of .str.encode() and .str.decode() methods (GH13008)
.str.encode()
.str.decode()
Improved performance of to_numeric if input is numeric dtype (GH12777)
to_numeric
Improved performance of sparse arithmetic with IntIndex (GH13036)
IntIndex
usecols parameter in pd.read_csv is now respected even when the lines of a CSV file are not even (GH12203)
pd.read_csv
Bug in groupby.transform(..) when axis=1 is specified with a non-monotonic ordered index (GH12713)
groupby.transform(..)
axis=1
Bug in Period and PeriodIndex creation raises KeyError if freq="Minute" is specified. Note that “Minute” freq is deprecated in v0.17.0, and recommended to use freq="T" instead (GH11854)
freq="Minute"
freq="T"
Bug in .resample(...).count() with a PeriodIndex always raising a TypeError (GH12774)
.resample(...).count()
Bug in .resample(...) with a PeriodIndex casting to a DatetimeIndex when empty (GH12868)
.resample(...)
Bug in .resample(...) with a PeriodIndex when resampling to an existing frequency (GH12770)
Bug in printing data which contains Period with different freq raises ValueError (GH12615)
freq
Bug in Series construction with Categorical and dtype='category' is specified (GH12574)
Categorical
dtype='category'
Bugs in concatenation with a coercible dtype was too aggressive, resulting in different dtypes in output formatting when an object was longer than display.max_rows (GH12411, GH12045, GH11594, GH10571, GH12211)
display.max_rows
Bug in float_format option with option not being validated as a callable. (GH12706)
float_format
Bug in GroupBy.filter when dropna=False and no groups fulfilled the criteria (GH12768)
GroupBy.filter
dropna=False
Bug in __name__ of .cum* functions (GH12021)
.cum*
Bug in .astype() of a Float64Inde/Int64Index to an Int64Index (GH12881)
.astype()
Float64Inde/Int64Index
Int64Index
Bug in round tripping an integer based index in .to_json()/.read_json() when orient='index' (the default) (GH12866)
.to_json()/.read_json()
orient='index'
Bug in plotting Categorical dtypes cause error when attempting stacked bar plot (GH13019)
Compat with >= numpy 1.11 for NaT comparisons (GH12969)
NaT
Bug in .drop() with a non-unique MultiIndex. (GH12701)
.drop()
Bug in .concat of datetime tz-aware and naive DataFrames (GH12467)
.concat
Bug in correctly raising a ValueError in .resample(..).fillna(..) when passing a non-string (GH12952)
.resample(..).fillna(..)
Bug fixes in various encoding and header processing issues in pd.read_sas() (GH12659, GH12654, GH12647, GH12809)
pd.read_sas()
Bug in pd.crosstab() where would silently ignore aggfunc if values=None (GH12569).
aggfunc
values=None
Potential segfault in DataFrame.to_json when serialising datetime.time (GH11473).
DataFrame.to_json
datetime.time
Potential segfault in DataFrame.to_json when attempting to serialise 0d array (GH11299).
Segfault in to_json when attempting to serialise a DataFrame or Series with non-ndarray values; now supports serialization of category, sparse, and datetime64[ns, tz] dtypes (GH10778).
to_json
datetime64[ns, tz]
Bug in DataFrame.to_json with unsupported dtype not passed to default handler (GH12554).
Bug in .align not returning the sub-class (GH12983)
.align
Bug in aligning a Series with a DataFrame (GH13037)
Bug in ABCPanel in which Panel4D was not being considered as a valid instance of this generic type (GH12810)
ABCPanel
Panel4D
Bug in consistency of .name on .groupby(..).apply(..) cases (GH12363)
.name
.groupby(..).apply(..)
Bug in Timestamp.__repr__ that caused pprint to fail in nested structures (GH12622)
Timestamp.__repr__
pprint
Bug in Timedelta.min and Timedelta.max, the properties now report the true minimum/maximum timedeltas as recognized by pandas. See the documentation. (GH12727)
Timedelta.min
Timedelta.max
timedeltas
Bug in .quantile() with interpolation may coerce to float unexpectedly (GH12772)
.quantile()
float
Bug in .quantile() with empty Series may return scalar rather than empty Series (GH12772)
Bug in .loc with out-of-bounds in a large indexer would raise IndexError rather than KeyError (GH12527)
.loc
Bug in resampling when using a TimedeltaIndex and .asfreq(), would previously not include the final fencepost (GH12926)
.asfreq()
Bug in equality testing with a Categorical in a DataFrame (GH12564)
Bug in GroupBy.first(), .last() returns incorrect row when TimeGrouper is used (GH7453)
GroupBy.first()
.last()
TimeGrouper
Bug in pd.read_csv() with the c engine when specifying skiprows with newlines in quoted items (GH10911, GH12775)
skiprows
Bug in DataFrame timezone lost when assigning tz-aware datetime Series with alignment (GH12981)
Bug in .value_counts() when normalize=True and dropna=True where nulls still contributed to the normalized count (GH12558)
.value_counts()
normalize=True
dropna=True
Bug in Series.value_counts() loses name if its dtype is category (GH12835)
Series.value_counts()
Bug in Series.value_counts() loses timezone info (GH12835)
Bug in Series.value_counts(normalize=True) with Categorical raises UnboundLocalError (GH12835)
Series.value_counts(normalize=True)
UnboundLocalError
Bug in Panel.fillna() ignoring inplace=True (GH12633)
Panel.fillna()
inplace=True
Bug in pd.read_csv() when specifying names, usecols, and parse_dates simultaneously with the c engine (GH9755)
names
Bug in pd.read_csv() when specifying delim_whitespace=True and lineterminator simultaneously with the c engine (GH12912)
lineterminator
Bug in Series.rename, DataFrame.rename and DataFrame.rename_axis not treating Series as mappings to relabel (GH12623).
Series.rename
DataFrame.rename
DataFrame.rename_axis
Clean in .rolling.min and .rolling.max to enhance dtype handling (GH12373)
.rolling.min
.rolling.max
Bug in groupby where complex types are coerced to float (GH12902)
groupby
Bug in Series.map raises TypeError if its dtype is category or tz-aware datetime (GH12473)
Series.map
datetime
Bugs on 32bit platforms for some test comparisons (GH12972)
Bug in index coercion when falling back from RangeIndex construction (GH12893)
Better error message in window functions when invalid argument (e.g. a float window) is passed (GH12669)
Bug in slicing subclassed DataFrame defined to return subclassed Series may return normal Series (GH11559)
Bug in .str accessor methods may raise ValueError if input has name and the result is DataFrame or MultiIndex (GH12617)
.str
name
Bug in DataFrame.last_valid_index() and DataFrame.first_valid_index() on empty frames (GH12800)
DataFrame.last_valid_index()
DataFrame.first_valid_index()
Bug in CategoricalIndex.get_loc returns different result from regular Index (GH12531)
CategoricalIndex.get_loc
Bug in PeriodIndex.resample where name not propagated (GH12769)
PeriodIndex.resample
Bug in date_range closed keyword and timezones (GH12684).
date_range
closed
Bug in pd.concat raises AttributeError when input data contains tz-aware datetime and timedelta (GH12620)
pd.concat
Bug in pd.concat did not handle empty Series properly (GH11082)
Bug in .plot.bar alignment when width is specified with int (GH12979)
.plot.bar
width
int
Bug in fill_value is ignored if the argument to a binary operator is a constant (GH12723)
Bug in pd.read_html() when using bs4 flavor and parsing table with a header and only one column (GH9178)
pd.read_html()
Bug in .pivot_table when margins=True and dropna=True where nulls still contributed to margin count (GH12577)
.pivot_table
margins=True
Bug in .pivot_table when dropna=False where table index/column names disappear (GH12133)
Bug in pd.crosstab() when margins=True and dropna=False which raised (GH12642)
Bug in Series.name when name attribute can be a hashable type (GH12610)
Series.name
Bug in .describe() resets categorical columns information (GH11558)
.describe()
Bug where loffset argument was not applied when calling resample().count() on a timeseries (GH12725)
loffset
resample().count()
pd.read_excel() now accepts column names associated with keyword argument names (GH12870)
Bug in pd.to_numeric() with Index returns np.ndarray, rather than Index (GH12777)
pd.to_numeric()
np.ndarray
Bug in pd.to_numeric() with datetime-like may raise TypeError (GH12777)
Bug in pd.to_numeric() with scalar raises ValueError (GH12777)
A total of 60 people contributed patches to this release. People with a “+” by their names contributed a patch for the first time.
Andrew Fiore-Gartland +
Bastiaan +
Benoît Vinot +
Brandon Rhodes +
DaCoEx +
Drew Fustin +
Ernesto Freitas +
Filip Ter +
Gregory Livschitz +
Gábor Lipták
Hassan Kibirige +
Iblis Lin
Israel Saeta Pérez +
Jason Wolosonovich +
Jeff Reback
Joe Jevnik
Joris Van den Bossche
Joshua Storck +
Ka Wo Chen
Kerby Shedden
Kieran O’Mahony
Leif Walsh +
Mahmoud Lababidi +
Maoyuan Liu +
Mark Roth +
Matt Wittmann
MaxU +
Maximilian Roos
Michael Droettboom +
Nick Eubank
Nicolas Bonnotte
OXPHOS +
Pauli Virtanen +
Peter Waller +
Pietro Battiston
Prabhjot Singh +
Robin Wilson
Roger Thomas +
Sebastian Bank
Stephen Hoover
Tim Hopper +
Tom Augspurger
WANG Aiyong
Wes Turner
Winand +
Xbar +
Yan Facai +
adneu +
ajenkins-cargometrics +
behzad nouri
chinskiy +
gfyoung
jeps-journal +
jonaslb +
kotrfa +
nileracecrew +
onesandzeroes
rs2 +
sinhrks
tsdlovell +