Pandas provides data structures for efficiently storing sparse data. These are not necessarily sparse in the typical “mostly 0”. Rather, you can view these objects as being “compressed” where any data matching a specific value (NaN / missing value, though any value can be chosen, including 0) is omitted. The compressed values are not actually stored in the array.
NaN
In [1]: arr = np.random.randn(10) In [2]: arr[2:-2] = np.nan In [3]: ts = pd.Series(pd.arrays.SparseArray(arr)) In [4]: ts Out[4]: 0 0.469112 1 -0.282863 2 NaN 3 NaN 4 NaN 5 NaN 6 NaN 7 NaN 8 -0.861849 9 -2.104569 dtype: Sparse[float64, nan]
Notice the dtype, Sparse[float64, nan]. The nan means that elements in the array that are nan aren’t actually stored, only the non-nan elements are. Those non-nan elements have a float64 dtype.
Sparse[float64, nan]
nan
float64
The sparse objects exist for memory efficiency reasons. Suppose you had a large, mostly NA DataFrame:
DataFrame
In [5]: df = pd.DataFrame(np.random.randn(10000, 4)) In [6]: df.iloc[:9998] = np.nan In [7]: sdf = df.astype(pd.SparseDtype("float", np.nan)) In [8]: sdf.head() Out[8]: 0 1 2 3 0 NaN NaN NaN NaN 1 NaN NaN NaN NaN 2 NaN NaN NaN NaN 3 NaN NaN NaN NaN 4 NaN NaN NaN NaN In [9]: sdf.dtypes Out[9]: 0 Sparse[float64, nan] 1 Sparse[float64, nan] 2 Sparse[float64, nan] 3 Sparse[float64, nan] dtype: object In [10]: sdf.sparse.density Out[10]: 0.0002
As you can see, the density (% of values that have not been “compressed”) is extremely low. This sparse object takes up much less memory on disk (pickled) and in the Python interpreter.
In [11]: 'dense : {:0.2f} bytes'.format(df.memory_usage().sum() / 1e3) Out[11]: 'dense : 320.13 bytes' In [12]: 'sparse: {:0.2f} bytes'.format(sdf.memory_usage().sum() / 1e3) Out[12]: 'sparse: 0.22 bytes'
Functionally, their behavior should be nearly identical to their dense counterparts.
arrays.SparseArray is a ExtensionArray for storing an array of sparse values (see dtypes for more on extension arrays). It is a 1-dimensional ndarray-like object storing only values distinct from the fill_value:
arrays.SparseArray
ExtensionArray
fill_value
In [13]: arr = np.random.randn(10) In [14]: arr[2:5] = np.nan In [15]: arr[7:8] = np.nan In [16]: sparr = pd.arrays.SparseArray(arr) In [17]: sparr Out[17]: [-1.9556635297215477, -1.6588664275960427, nan, nan, nan, 1.1589328886422277, 0.14529711373305043, nan, 0.6060271905134522, 1.3342113401317768] Fill: nan IntIndex Indices: array([0, 1, 5, 6, 8, 9], dtype=int32)
A sparse array can be converted to a regular (dense) ndarray with numpy.asarray()
numpy.asarray()
In [18]: np.asarray(sparr) Out[18]: array([-1.9557, -1.6589, nan, nan, nan, 1.1589, 0.1453, nan, 0.606 , 1.3342])
The SparseArray.dtype property stores two pieces of information
SparseArray.dtype
The dtype of the non-sparse values
The scalar fill value
In [19]: sparr.dtype Out[19]: Sparse[float64, nan]
A SparseDtype may be constructed by passing each of these
SparseDtype
In [20]: pd.SparseDtype(np.dtype('datetime64[ns]')) Out[20]: Sparse[datetime64[ns], NaT]
The default fill value for a given NumPy dtype is the “missing” value for that dtype, though it may be overridden.
In [21]: pd.SparseDtype(np.dtype('datetime64[ns]'), ....: fill_value=pd.Timestamp('2017-01-01')) ....: Out[21]: Sparse[datetime64[ns], 2017-01-01 00:00:00]
Finally, the string alias 'Sparse[dtype]' may be used to specify a sparse dtype in many places
'Sparse[dtype]'
In [22]: pd.array([1, 0, 0, 2], dtype='Sparse[int]') Out[22]: [1, 0, 0, 2] Fill: 0 IntIndex Indices: array([0, 3], dtype=int32)
New in version 0.24.0.
Pandas provides a .sparse accessor, similar to .str for string data, .cat for categorical data, and .dt for datetime-like data. This namespace provides attributes and methods that are specific to sparse data.
.sparse
.str
.cat
.dt
In [23]: s = pd.Series([0, 0, 1, 2], dtype="Sparse[int]") In [24]: s.sparse.density Out[24]: 0.5 In [25]: s.sparse.fill_value Out[25]: 0
This accessor is available only on data with SparseDtype, and on the Series class itself for creating a Series with sparse data from a scipy COO matrix with.
Series
New in version 0.25.0.
A .sparse accessor has been added for DataFrame as well. See Sparse accessor for more.
You can apply NumPy ufuncs to SparseArray and get a SparseArray as a result.
SparseArray
In [26]: arr = pd.arrays.SparseArray([1., np.nan, np.nan, -2., np.nan]) In [27]: np.abs(arr) Out[27]: [1.0, nan, nan, 2.0, nan] Fill: nan IntIndex Indices: array([0, 3], dtype=int32)
The ufunc is also applied to fill_value. This is needed to get the correct dense result.
In [28]: arr = pd.arrays.SparseArray([1., -1, -1, -2., -1], fill_value=-1) In [29]: np.abs(arr) Out[29]: [1.0, 1, 1, 2.0, 1] Fill: 1 IntIndex Indices: array([0, 3], dtype=int32) In [30]: np.abs(arr).to_dense() Out[30]: array([1., 1., 1., 2., 1.])
Note
SparseSeries and SparseDataFrame were removed in pandas 1.0.0. This migration guide is present to aid in migrating from previous versions.
SparseSeries
SparseDataFrame
In older versions of pandas, the SparseSeries and SparseDataFrame classes (documented below) were the preferred way to work with sparse data. With the advent of extension arrays, these subclasses are no longer needed. Their purpose is better served by using a regular Series or DataFrame with sparse values instead.
There’s no performance or memory penalty to using a Series or DataFrame with sparse values, rather than a SparseSeries or SparseDataFrame.
This section provides some guidance on migrating your code to the new style. As a reminder, you can use the python warnings module to control warnings. But we recommend modifying your code, rather than ignoring the warning.
Construction
From an array-like, use the regular Series or DataFrame constructors with SparseArray values.
# Previous way >>> pd.SparseDataFrame({"A": [0, 1]})
# New way In [31]: pd.DataFrame({"A": pd.arrays.SparseArray([0, 1])}) Out[31]: A 0 0 1 1
From a SciPy sparse matrix, use DataFrame.sparse.from_spmatrix(),
DataFrame.sparse.from_spmatrix()
# Previous way >>> from scipy import sparse >>> mat = sparse.eye(3) >>> df = pd.SparseDataFrame(mat, columns=['A', 'B', 'C'])
# New way In [32]: from scipy import sparse --------------------------------------------------------------------------- ModuleNotFoundError Traceback (most recent call last) <ipython-input-32-cd6a00615037> in <module> ----> 1 from scipy import sparse ModuleNotFoundError: No module named 'scipy' In [33]: mat = sparse.eye(3) --------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-33-b8a897596f66> in <module> ----> 1 mat = sparse.eye(3) NameError: name 'sparse' is not defined In [34]: df = pd.DataFrame.sparse.from_spmatrix(mat, columns=['A', 'B', 'C']) --------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-34-7bac7a79b916> in <module> ----> 1 df = pd.DataFrame.sparse.from_spmatrix(mat, columns=['A', 'B', 'C']) NameError: name 'mat' is not defined In [35]: df.dtypes Out[35]: 0 float64 1 float64 2 float64 3 float64 dtype: object
Conversion
From sparse to dense, use the .sparse accessors
In [36]: df.sparse.to_dense() --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-36-6c28aed8f0df> in <module> ----> 1 df.sparse.to_dense() ~/scipy/pandas/pandas/core/generic.py in __getattr__(self, name) 5144 or name in self._accessors 5145 ): -> 5146 return object.__getattribute__(self, name) 5147 else: 5148 if self._info_axis._can_hold_identifiers_and_holds_name(name): ~/scipy/pandas/pandas/core/accessor.py in __get__(self, obj, cls) 185 # we're accessing the attribute of the class, i.e., Dataset.geo 186 return self._accessor --> 187 accessor_obj = self._accessor(obj) 188 # Replace the property with the accessor object. Inspired by: 189 # http://www.pydanny.com/cached-property.html ~/scipy/pandas/pandas/core/arrays/sparse/accessor.py in __init__(self, data) 17 def __init__(self, data=None): 18 self._parent = data ---> 19 self._validate(data) 20 21 def _validate(self, data): ~/scipy/pandas/pandas/core/arrays/sparse/accessor.py in _validate(self, data) 195 dtypes = data.dtypes 196 if not all(isinstance(t, SparseDtype) for t in dtypes): --> 197 raise AttributeError(self._validation_msg) 198 199 @classmethod AttributeError: Can only use the '.sparse' accessor with Sparse data. In [37]: df.sparse.to_coo() --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-37-4c6c08d2fee9> in <module> ----> 1 df.sparse.to_coo() ~/scipy/pandas/pandas/core/generic.py in __getattr__(self, name) 5144 or name in self._accessors 5145 ): -> 5146 return object.__getattribute__(self, name) 5147 else: 5148 if self._info_axis._can_hold_identifiers_and_holds_name(name): ~/scipy/pandas/pandas/core/accessor.py in __get__(self, obj, cls) 185 # we're accessing the attribute of the class, i.e., Dataset.geo 186 return self._accessor --> 187 accessor_obj = self._accessor(obj) 188 # Replace the property with the accessor object. Inspired by: 189 # http://www.pydanny.com/cached-property.html ~/scipy/pandas/pandas/core/arrays/sparse/accessor.py in __init__(self, data) 17 def __init__(self, data=None): 18 self._parent = data ---> 19 self._validate(data) 20 21 def _validate(self, data): ~/scipy/pandas/pandas/core/arrays/sparse/accessor.py in _validate(self, data) 195 dtypes = data.dtypes 196 if not all(isinstance(t, SparseDtype) for t in dtypes): --> 197 raise AttributeError(self._validation_msg) 198 199 @classmethod AttributeError: Can only use the '.sparse' accessor with Sparse data.
From dense to sparse, use DataFrame.astype() with a SparseDtype.
DataFrame.astype()
In [38]: dense = pd.DataFrame({"A": [1, 0, 0, 1]}) In [39]: dtype = pd.SparseDtype(int, fill_value=0) In [40]: dense.astype(dtype) Out[40]: A 0 1 1 0 2 0 3 1
Sparse Properties
Sparse-specific properties, like density, are available on the .sparse accessor.
density
In [41]: df.sparse.density --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-41-a2836336c5bf> in <module> ----> 1 df.sparse.density ~/scipy/pandas/pandas/core/generic.py in __getattr__(self, name) 5144 or name in self._accessors 5145 ): -> 5146 return object.__getattribute__(self, name) 5147 else: 5148 if self._info_axis._can_hold_identifiers_and_holds_name(name): ~/scipy/pandas/pandas/core/accessor.py in __get__(self, obj, cls) 185 # we're accessing the attribute of the class, i.e., Dataset.geo 186 return self._accessor --> 187 accessor_obj = self._accessor(obj) 188 # Replace the property with the accessor object. Inspired by: 189 # http://www.pydanny.com/cached-property.html ~/scipy/pandas/pandas/core/arrays/sparse/accessor.py in __init__(self, data) 17 def __init__(self, data=None): 18 self._parent = data ---> 19 self._validate(data) 20 21 def _validate(self, data): ~/scipy/pandas/pandas/core/arrays/sparse/accessor.py in _validate(self, data) 195 dtypes = data.dtypes 196 if not all(isinstance(t, SparseDtype) for t in dtypes): --> 197 raise AttributeError(self._validation_msg) 198 199 @classmethod AttributeError: Can only use the '.sparse' accessor with Sparse data.
General differences
In a SparseDataFrame, all columns were sparse. A DataFrame can have a mixture of sparse and dense columns. As a consequence, assigning new columns to a DataFrame with sparse values will not automatically convert the input to be sparse.
# Previous Way >>> df = pd.SparseDataFrame({"A": [0, 1]}) >>> df['B'] = [0, 0] # implicitly becomes Sparse >>> df['B'].dtype Sparse[int64, nan]
Instead, you’ll need to ensure that the values being assigned are sparse
In [42]: df = pd.DataFrame({"A": pd.arrays.SparseArray([0, 1])}) In [43]: df['B'] = [0, 0] # remains dense In [44]: df['B'].dtype Out[44]: dtype('int64') In [45]: df['B'] = pd.arrays.SparseArray([0, 0]) In [46]: df['B'].dtype Out[46]: Sparse[int64, 0]
The SparseDataFrame.default_kind and SparseDataFrame.default_fill_value attributes have no replacement.
SparseDataFrame.default_kind
SparseDataFrame.default_fill_value
Use DataFrame.sparse.from_spmatrix() to create a DataFrame with sparse values from a sparse matrix.
In [47]: from scipy.sparse import csr_matrix --------------------------------------------------------------------------- ModuleNotFoundError Traceback (most recent call last) <ipython-input-47-4a61a219f322> in <module> ----> 1 from scipy.sparse import csr_matrix ModuleNotFoundError: No module named 'scipy' In [48]: arr = np.random.random(size=(1000, 5)) In [49]: arr[arr < .9] = 0 In [50]: sp_arr = csr_matrix(arr) --------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-50-318e903374d9> in <module> ----> 1 sp_arr = csr_matrix(arr) NameError: name 'csr_matrix' is not defined In [51]: sp_arr --------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-51-b1055a0a0153> in <module> ----> 1 sp_arr NameError: name 'sp_arr' is not defined In [52]: sdf = pd.DataFrame.sparse.from_spmatrix(sp_arr) --------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-52-6cb924044411> in <module> ----> 1 sdf = pd.DataFrame.sparse.from_spmatrix(sp_arr) NameError: name 'sp_arr' is not defined In [53]: sdf.head() Out[53]: 0 1 2 3 0 NaN NaN NaN NaN 1 NaN NaN NaN NaN 2 NaN NaN NaN NaN 3 NaN NaN NaN NaN 4 NaN NaN NaN NaN In [54]: sdf.dtypes Out[54]: 0 Sparse[float64, nan] 1 Sparse[float64, nan] 2 Sparse[float64, nan] 3 Sparse[float64, nan] dtype: object
All sparse formats are supported, but matrices that are not in COOrdinate format will be converted, copying data as needed. To convert back to sparse SciPy matrix in COO format, you can use the DataFrame.sparse.to_coo() method:
COOrdinate
DataFrame.sparse.to_coo()
In [55]: sdf.sparse.to_coo() --------------------------------------------------------------------------- ImportError Traceback (most recent call last) <ipython-input-55-139b822f8f1b> in <module> ----> 1 sdf.sparse.to_coo() ~/scipy/pandas/pandas/core/arrays/sparse/accessor.py in to_coo(self) 285 and uint64 will result in a float64 dtype. 286 """ --> 287 import_optional_dependency("scipy") 288 from scipy.sparse import coo_matrix 289 ~/scipy/pandas/pandas/compat/_optional.py in import_optional_dependency(name, extra, raise_on_missing, on_version) 89 except ImportError: 90 if raise_on_missing: ---> 91 raise ImportError(msg) from None 92 else: 93 return None ImportError: Missing optional dependency 'scipy'. Use pip or conda to install scipy.
meth:Series.sparse.to_coo is implemented for transforming a Series with sparse values indexed by a MultiIndex to a scipy.sparse.coo_matrix.
MultiIndex
scipy.sparse.coo_matrix
The method requires a MultiIndex with two or more levels.
In [56]: s = pd.Series([3.0, np.nan, 1.0, 3.0, np.nan, np.nan]) In [57]: s.index = pd.MultiIndex.from_tuples([(1, 2, 'a', 0), ....: (1, 2, 'a', 1), ....: (1, 1, 'b', 0), ....: (1, 1, 'b', 1), ....: (2, 1, 'b', 0), ....: (2, 1, 'b', 1)], ....: names=['A', 'B', 'C', 'D']) ....: In [58]: s Out[58]: A B C D 1 2 a 0 3.0 1 NaN 1 b 0 1.0 1 3.0 2 1 b 0 NaN 1 NaN dtype: float64 In [59]: ss = s.astype('Sparse') In [60]: ss Out[60]: A B C D 1 2 a 0 3.0 1 NaN 1 b 0 1.0 1 3.0 2 1 b 0 NaN 1 NaN dtype: Sparse[float64, nan]
In the example below, we transform the Series to a sparse representation of a 2-d array by specifying that the first and second MultiIndex levels define labels for the rows and the third and fourth levels define labels for the columns. We also specify that the column and row labels should be sorted in the final sparse representation.
In [61]: A, rows, columns = ss.sparse.to_coo(row_levels=['A', 'B'], ....: column_levels=['C', 'D'], ....: sort_labels=True) ....: --------------------------------------------------------------------------- ModuleNotFoundError Traceback (most recent call last) <ipython-input-61-6a0432cd14ff> in <module> 1 A, rows, columns = ss.sparse.to_coo(row_levels=['A', 'B'], 2 column_levels=['C', 'D'], ----> 3 sort_labels=True) ~/scipy/pandas/pandas/core/arrays/sparse/accessor.py in to_coo(self, row_levels, column_levels, sort_labels) 146 147 A, rows, columns = _sparse_series_to_coo( --> 148 self._parent, row_levels, column_levels, sort_labels=sort_labels 149 ) 150 return A, rows, columns ~/scipy/pandas/pandas/core/arrays/sparse/scipy_sparse.py in _sparse_series_to_coo(ss, row_levels, column_levels, sort_labels) 91 """ 92 ---> 93 import scipy.sparse 94 95 if ss.index.nlevels < 2: ModuleNotFoundError: No module named 'scipy' In [62]: A --------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-62-7d157d7c000a> in <module> ----> 1 A NameError: name 'A' is not defined In [63]: A.todense() --------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-63-a3fb4a959fae> in <module> ----> 1 A.todense() NameError: name 'A' is not defined In [64]: rows --------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-64-870319074abe> in <module> ----> 1 rows NameError: name 'rows' is not defined In [65]: columns Out[65]: ['id_0', 'name_0', 'x_0', 'y_0']
Specifying different row and column labels (and not sorting them) yields a different sparse matrix:
In [66]: A, rows, columns = ss.sparse.to_coo(row_levels=['A', 'B', 'C'], ....: column_levels=['D'], ....: sort_labels=False) ....: --------------------------------------------------------------------------- ModuleNotFoundError Traceback (most recent call last) <ipython-input-66-e5cb088494bb> in <module> 1 A, rows, columns = ss.sparse.to_coo(row_levels=['A', 'B', 'C'], 2 column_levels=['D'], ----> 3 sort_labels=False) ~/scipy/pandas/pandas/core/arrays/sparse/accessor.py in to_coo(self, row_levels, column_levels, sort_labels) 146 147 A, rows, columns = _sparse_series_to_coo( --> 148 self._parent, row_levels, column_levels, sort_labels=sort_labels 149 ) 150 return A, rows, columns ~/scipy/pandas/pandas/core/arrays/sparse/scipy_sparse.py in _sparse_series_to_coo(ss, row_levels, column_levels, sort_labels) 91 """ 92 ---> 93 import scipy.sparse 94 95 if ss.index.nlevels < 2: ModuleNotFoundError: No module named 'scipy' In [67]: A --------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-67-7d157d7c000a> in <module> ----> 1 A NameError: name 'A' is not defined In [68]: A.todense() --------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-68-a3fb4a959fae> in <module> ----> 1 A.todense() NameError: name 'A' is not defined In [69]: rows --------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-69-870319074abe> in <module> ----> 1 rows NameError: name 'rows' is not defined In [70]: columns Out[70]: ['id_0', 'name_0', 'x_0', 'y_0']
A convenience method Series.sparse.from_coo() is implemented for creating a Series with sparse values from a scipy.sparse.coo_matrix.
Series.sparse.from_coo()
In [71]: from scipy import sparse --------------------------------------------------------------------------- ModuleNotFoundError Traceback (most recent call last) <ipython-input-71-cd6a00615037> in <module> ----> 1 from scipy import sparse ModuleNotFoundError: No module named 'scipy' In [72]: A = sparse.coo_matrix(([3.0, 1.0, 2.0], ([1, 0, 0], [0, 2, 3])), ....: shape=(3, 4)) ....: --------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-72-22b6f5faecd2> in <module> ----> 1 A = sparse.coo_matrix(([3.0, 1.0, 2.0], ([1, 0, 0], [0, 2, 3])), 2 shape=(3, 4)) NameError: name 'sparse' is not defined In [73]: A --------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-73-7d157d7c000a> in <module> ----> 1 A NameError: name 'A' is not defined In [74]: A.todense() --------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-74-a3fb4a959fae> in <module> ----> 1 A.todense() NameError: name 'A' is not defined
The default behaviour (with dense_index=False) simply returns a Series containing only the non-null entries.
dense_index=False
In [75]: ss = pd.Series.sparse.from_coo(A) --------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-75-5caf58835db8> in <module> ----> 1 ss = pd.Series.sparse.from_coo(A) NameError: name 'A' is not defined In [76]: ss Out[76]: A B C D 1 2 a 0 3.0 1 NaN 1 b 0 1.0 1 3.0 2 1 b 0 NaN 1 NaN dtype: Sparse[float64, nan]
Specifying dense_index=True will result in an index that is the Cartesian product of the row and columns coordinates of the matrix. Note that this will consume a significant amount of memory (relative to dense_index=False) if the sparse matrix is large (and sparse) enough.
dense_index=True
In [77]: ss_dense = pd.Series.sparse.from_coo(A, dense_index=True) --------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-77-245bf04182d5> in <module> ----> 1 ss_dense = pd.Series.sparse.from_coo(A, dense_index=True) NameError: name 'A' is not defined In [78]: ss_dense --------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-78-d9f97589c3d3> in <module> ----> 1 ss_dense NameError: name 'ss_dense' is not defined