Contrib reference¶
Commonly used helper functions, to support in the setup of specific transformations.
contrib.read¶
-
data_migrator.contrib.read.
read_map_from_csv
(key=0, value=1, f=None, delimiter='\t', header=True, as_list=False, first=True, unique=False)[source]¶ Generates a map from a csv and adds some validation and list parsing. A function that returns a map for MappingField to use as input in its MappingField.data_map.
>>> from data_migrator.contrib.read import read_map_from_csv >>> table_map = read_map_from_csv(f=open('table.csv'), delimiter=';', key='id', value='name') >>> len(table_map) 10
Note that by default it is expected to have headers in the csv.
Parameters: - f – Filehandle to read the csv from into the map
- delimiter – Option to select another delimiter, other than \t
- key – Name or position of the Key, if
header
is false, the ordinal position is expected (default first) - value – Name or position of the Value, if
header
is false, the ordinal position is expected (default second) - as_list (boolean) – If
True
, data-migrator will treat add all values forkey
as a list. Default isFalse
. - first (boolean) – If
True
, data-migrator will keep first if non unique values forkey
. Default isTrue
. - unique (boolean) – If
True
, data-migrator will treat add all non unique values forkey
as a violation and raise aNonUniqueDataException
. Default isFalse
. - header (boolean) – If
True
, data-migrator will treat row as a header column. Default isTrue
Returns: a key, value map from the csv
Return type: map
Raises: DefinitionException
– if key, value does not match or as_list not set.NonUniqueDataException
– if data is not unique on the key.
-
data_migrator.contrib.read.
read_model_from_csv
(model, f=None, delimiter='\t', header=True)[source]¶ Reads a model from a csv.
>>> from data_migrator.contrib.read import read_model_from_csv >>> read_map_from_csv(model=model.TestModel, f=open('table.csv'), delimiter=';'), >>> len(model.TestModel) 10
Note that by default it is expected to have headers in the csv.
Parameters: - model – reference to Model to read
- f – Filehandle to read the csv from into the map
- delimiter – Option to select another delimiter, other than \t
- header (boolean) – If
True
, data-migrator will treat row as a header column. Default isTrue
Returns: None, but Model.objects will contain the read records
contrib.dutch¶
commonly used dutch support functions for cleaning and anonymization
-
class
data_migrator.contrib.dutch.
PhoneAnonymizor
[source]¶ PhoneAnonymizor generates a random dutch phonenumber, like +3142097272
>>> len(PhoneAnonymizor()('020-1234583')) 11 >>> len(PhoneAnonymizor()('06-12345678')) 11
-
class
data_migrator.contrib.dutch.
ZipCodeAnonymizor
[source]¶ ZipCodeAnonymizor generates a random dutch zipcode, like ‘4897 LD’
>>> len(ZipCodeAnonymizor()('1234 aa')) 7
-
data_migrator.contrib.dutch.
clean_phone
(v)[source]¶ Cleans phone numbers to dutch clean format
clean_phone clean phone numbers, replaces all characters and spaces adds dutch country code (+31) if no country code is provide
>>> clean_phone('00 31 6 - 20 20 20 20') '+31620202020' >>> clean_phone('06 20 20 20 20') '+31620202020' >>> clean_phone('020 -123 345 6') '+31201233456' >>> clean_phone('+440.203.020.23') '+44020302023' >>> clean_phone('+440a203a020a23') '+44020302023' >>> clean_phone('+440 ada 203.020 // 23') '+44020302023' >>> clean_phone('31 (6) - 20 20 20 20') '+31620202020'
Parameters: v (str) – value to clean Returns: cleaned phone number Return type: str
-
data_migrator.contrib.dutch.
clean_zip_code
(v)[source]¶ Cleans a dutch zipcode
>>> clean_zip_code('1234 aa') '1234AA' >>> clean_zip_code('1234AB') '1234AB' >>> clean_zip_code('1234 Ba') '1234BA' >>> clean_zip_code('1 2 3 4 A B') '1 2 3 4 A B' >>> clean_zip_code('blabla') 'blabla'
Parameters: v (str) – zipcode to clean Returns: cleaned zip code Return type: str