This is a minor bug-fix release from 0.15.0 and includes a small number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. We recommend that all users upgrade to this version.
Enhancements
API Changes
Bug Fixes
s.dt.hour and other .dt accessors will now return np.nan for missing values (rather than previously -1), (GH8689)
s.dt.hour
.dt
np.nan
In [1]: s = pd.Series(pd.date_range('20130101', periods=5, freq='D')) In [2]: s.iloc[2] = np.nan In [3]: s Out[3]: 0 2013-01-01 1 2013-01-02 2 NaT 3 2013-01-04 4 2013-01-05 Length: 5, dtype: datetime64[ns]
previous behavior:
In [6]: s.dt.hour Out[6]: 0 0 1 0 2 -1 3 0 4 0 dtype: int64
current behavior:
In [4]: s.dt.hour Out[4]: 0 0.0 1 0.0 2 NaN 3 0.0 4 0.0 Length: 5, dtype: float64
groupby with as_index=False will not add erroneous extra columns to result (GH8582):
groupby
as_index=False
In [5]: np.random.seed(2718281) In [6]: df = pd.DataFrame(np.random.randint(0, 100, (10, 2)), ...: columns=['jim', 'joe']) ...: In [7]: df.head() Out[7]: jim joe 0 61 81 1 96 49 2 55 65 3 72 51 4 77 12 [5 rows x 2 columns] In [8]: ts = pd.Series(5 * np.random.randint(0, 3, 10))
In [4]: df.groupby(ts, as_index=False).max() Out[4]: NaN jim joe 0 0 72 83 1 5 77 84 2 10 96 65
In [9]: df.groupby(ts, as_index=False).max() Out[9]: jim joe 0 72 83 1 77 84 2 96 65 [3 rows x 2 columns]
groupby will not erroneously exclude columns if the column name conflicts with the grouper name (GH8112):
In [10]: df = pd.DataFrame({'jim': range(5), 'joe': range(5, 10)}) In [11]: df Out[11]: jim joe 0 0 5 1 1 6 2 2 7 3 3 8 4 4 9 [5 rows x 2 columns] In [12]: gr = df.groupby(df['jim'] < 2)
previous behavior (excludes 1st column from output):
In [4]: gr.apply(sum) Out[4]: joe jim False 24 True 11
In [13]: gr.apply(sum) Out[13]: jim joe jim False 9 24 True 1 11 [2 rows x 2 columns]
Support for slicing with monotonic decreasing indexes, even if start or stop is not found in the index (GH7860):
start
stop
In [14]: s = pd.Series(['a', 'b', 'c', 'd'], [4, 3, 2, 1]) In [15]: s Out[15]: 4 a 3 b 2 c 1 d Length: 4, dtype: object
In [8]: s.loc[3.5:1.5] KeyError: 3.5
In [16]: s.loc[3.5:1.5] Out[16]: 3 b 2 c Length: 2, dtype: object
io.data.Options has been fixed for a change in the format of the Yahoo Options page (GH8612), (GH8741)
io.data.Options
Note
As a result of a change in Yahoo’s option page layout, when an expiry date is given, Options methods now return data for a single expiry date. Previously, methods returned all data for the selected month.
Options
The month and year parameters have been undeprecated and can be used to get all options data for a given month.
month
year
If an expiry date that is not valid is given, data for the next expiry after the given date is returned.
Option data frames are now saved on the instance as callsYYMMDD or putsYYMMDD. Previously they were saved as callsMMYY and putsMMYY. The next expiry is saved as calls and puts.
callsYYMMDD
putsYYMMDD
callsMMYY
putsMMYY
calls
puts
New features:
The expiry parameter can now be a single date or a list-like object containing dates.
A new property expiry_dates was added, which returns all available expiry dates.
expiry_dates
Current behavior:
In [17]: from pandas.io.data import Options In [18]: aapl = Options('aapl', 'yahoo') In [19]: aapl.get_call_data().iloc[0:5, 0:1] Out[19]: Last Strike Expiry Type Symbol 80 2014-11-14 call AAPL141114C00080000 29.05 84 2014-11-14 call AAPL141114C00084000 24.80 85 2014-11-14 call AAPL141114C00085000 24.05 86 2014-11-14 call AAPL141114C00086000 22.76 87 2014-11-14 call AAPL141114C00087000 21.74 In [20]: aapl.expiry_dates Out[20]: [datetime.date(2014, 11, 14), datetime.date(2014, 11, 22), datetime.date(2014, 11, 28), datetime.date(2014, 12, 5), datetime.date(2014, 12, 12), datetime.date(2014, 12, 20), datetime.date(2015, 1, 17), datetime.date(2015, 2, 20), datetime.date(2015, 4, 17), datetime.date(2015, 7, 17), datetime.date(2016, 1, 15), datetime.date(2017, 1, 20)] In [21]: aapl.get_near_stock_price(expiry=aapl.expiry_dates[0:3]).iloc[0:5, 0:1] Out[21]: Last Strike Expiry Type Symbol 109 2014-11-22 call AAPL141122C00109000 1.48 2014-11-28 call AAPL141128C00109000 1.79 110 2014-11-14 call AAPL141114C00110000 0.55 2014-11-22 call AAPL141122C00110000 1.02 2014-11-28 call AAPL141128C00110000 1.32
pandas now also registers the datetime64 dtype in matplotlib’s units registry to plot such values as datetimes. This is activated once pandas is imported. In previous versions, plotting an array of datetime64 values will have resulted in plotted integer values. To keep the previous behaviour, you can do del matplotlib.units.registry[np.datetime64] (GH8614).
datetime64
del matplotlib.units.registry[np.datetime64]
concat permits a wider variety of iterables of pandas objects to be passed as the first parameter (GH8645):
concat
In [17]: from collections import deque In [18]: df1 = pd.DataFrame([1, 2, 3]) In [19]: df2 = pd.DataFrame([4, 5, 6])
In [7]: pd.concat(deque((df1, df2))) TypeError: first argument must be a list-like of pandas objects, you passed an object of type "deque"
In [20]: pd.concat(deque((df1, df2))) Out[20]: 0 0 1 1 2 2 3 0 4 1 5 2 6 [6 rows x 1 columns]
Represent MultiIndex labels with a dtype that utilizes memory based on the level size. In prior versions, the memory usage was a constant 8 bytes per element in each level. In addition, in prior versions, the reported memory usage was incorrect as it didn’t show the usage for the memory occupied by the underling data array. (GH8456)
MultiIndex
In [21]: dfi = pd.DataFrame(1, index=pd.MultiIndex.from_product([['a'], ....: range(1000)]), columns=['A']) ....:
# this was underreported in prior versions In [1]: dfi.memory_usage(index=True) Out[1]: Index 8000 # took about 24008 bytes in < 0.15.1 A 8000 dtype: int64
In [22]: dfi.memory_usage(index=True) Out[22]: Index 11040 A 8000 Length: 2, dtype: int64
Added Index properties is_monotonic_increasing and is_monotonic_decreasing (GH8680).
Added option to select columns when importing Stata files (GH7935)
Qualify memory usage in DataFrame.info() by adding + if it is a lower bound (GH8578)
DataFrame.info()
+
Raise errors in certain aggregation cases where an argument such as numeric_only is not handled (GH8592).
numeric_only
Added support for 3-character ISO and non-standard country codes in io.wb.download() (GH8482)
io.wb.download()
World Bank data requests now will warn/raise based on an errors argument, as well as a list of hard-coded country codes and the World Bank’s JSON response. In prior versions, the error messages didn’t look at the World Bank’s JSON response. Problem-inducing input were simply dropped prior to the request. The issue was that many good countries were cropped in the hard-coded approach. All countries will work now, but some bad countries will raise exceptions because some edge cases break the entire response. (GH8482)
errors
Added option to Series.str.split() to return a DataFrame rather than a Series (GH8428)
Series.str.split()
DataFrame
Series
Added option to df.info(null_counts=None|True|False) to override the default display options and force showing of the null-counts (GH8701)
df.info(null_counts=None|True|False)
Bug in unpickling of a CustomBusinessDay object (GH8591)
CustomBusinessDay
Bug in coercing Categorical to a records array, e.g. df.to_records() (GH8626)
Categorical
df.to_records()
Bug in Categorical not created properly with Series.to_frame() (GH8626)
Series.to_frame()
Bug in coercing in astype of a Categorical of a passed pd.Categorical (this now raises TypeError correctly), (GH8626)
pd.Categorical
TypeError
Bug in cut/qcut when using Series and retbins=True (GH8589)
cut
qcut
retbins=True
Bug in writing Categorical columns to an SQL database with to_sql (GH8624).
to_sql
Bug in comparing Categorical of datetime raising when being compared to a scalar datetime (GH8687)
Bug in selecting from a Categorical with .iloc (GH8623)
.iloc
Bug in groupby-transform with a Categorical (GH8623)
Bug in duplicated/drop_duplicates with a Categorical (GH8623)
Bug in Categorical reflected comparison operator raising if the first argument was a numpy array scalar (e.g. np.int64) (GH8658)
Bug in Panel indexing with a list-like (GH8710)
Compat issue is DataFrame.dtypes when options.mode.use_inf_as_null is True (GH8722)
DataFrame.dtypes
options.mode.use_inf_as_null
Bug in read_csv, dialect parameter would not take a string (GH8703)
read_csv
dialect
Bug in slicing a MultiIndex level with an empty-list (GH8737)
Bug in numeric index operations of add/sub with Float/Index Index with numpy arrays (GH8608)
Bug in setitem with empty indexer and unwanted coercion of dtypes (GH8669)
Bug in ix/loc block splitting on setitem (manifests with integer-like dtypes, e.g. datetime64) (GH8607)
Bug when doing label based indexing with integers not found in the index for non-unique but monotonic indexes (GH8680).
Bug when indexing a Float64Index with np.nan on numpy 1.7 (GH8980).
Fix shape attribute for MultiIndex (GH8609)
shape
Bug in GroupBy where a name conflict between the grouper and columns would break groupby operations (GH7115, GH8112)
GroupBy
Fixed a bug where plotting a column y and specifying a label would mutate the index name of the original DataFrame (GH8494)
y
Fix regression in plotting of a DatetimeIndex directly with matplotlib (GH8614).
Bug in date_range where partially-specified dates would incorporate current date (GH6961)
date_range
Bug in Setting by indexer to a scalar value with a mixed-dtype Panel4d was failing (GH8702)
Bug where DataReader’s would fail if one of the symbols passed was invalid. Now returns data for valid symbols and np.nan for invalid (GH8494)
DataReader
Bug in get_quote_yahoo that wouldn’t allow non-float return values (GH5229).
get_quote_yahoo
A total of 23 people contributed patches to this release. People with a “+” by their names contributed a patch for the first time.
Aaron Staple +
Andrew Rosenfeld
Anton I. Sipos
Artemy Kolchinsky
Bill Letson +
Dave Hughes +
David Stephens
Guillaume Horel +
Jeff Reback
Joris Van den Bossche
Kevin Sheppard
Nick Stahl +
Sanghee Kim +
Stephan Hoyer
Tom Augspurger
TomAugspurger
WANG Aiyong +
behzad nouri
immerrr
jnmclarty
jreback
pallav-fdsi +
unutbu