Contrib reference

Commonly used helper functions, to support in the setup of specific transformations.

contrib.read

data_migrator.contrib.read.read_map_from_csv(key=0, value=1, f=None, delimiter='\t', header=True, as_list=False, first=True, unique=False)[source]

Generates a map from a csv and adds some validation and list parsing. A function that returns a map for MappingField to use as input in its MappingField.data_map.

>>> from data_migrator.contrib.read import read_map_from_csv
>>> table_map = read_map_from_csv(f=open('table.csv'), delimiter=';',
                                  key='id', value='name')
>>> len(table_map)
10

Note that by default it is expected to have headers in the csv.

Parameters:
  • f – Filehandle to read the csv from into the map
  • delimiter – Option to select another delimiter, other than \t
  • key – Name or position of the Key, if header is false, the ordinal position is expected (default first)
  • value – Name or position of the Value, if header is false, the ordinal position is expected (default second)
  • as_list (boolean) – If True, data-migrator will treat add all values for key as a list. Default is False.
  • first (boolean) – If True, data-migrator will keep first if non unique values for key. Default is True.
  • unique (boolean) – If True, data-migrator will treat add all non unique values for key as a violation and raise a NonUniqueDataException. Default is False.
  • header (boolean) – If True, data-migrator will treat row as a header column. Default is True
Returns:

a key, value map from the csv

Return type:

map

Raises:
data_migrator.contrib.read.read_model_from_csv(model, f=None, delimiter='\t', header=True)[source]

Reads a model from a csv.

>>> from data_migrator.contrib.read import read_model_from_csv
>>> read_map_from_csv(model=model.TestModel, f=open('table.csv'),
                      delimiter=';'),
>>> len(model.TestModel)
10

Note that by default it is expected to have headers in the csv.

Parameters:
  • model – reference to Model to read
  • f – Filehandle to read the csv from into the map
  • delimiter – Option to select another delimiter, other than \t
  • header (boolean) – If True, data-migrator will treat row as a header column. Default is True
Returns:

None, but Model.objects will contain the read records

contrib.dutch

commonly used dutch support functions for cleaning and anonymization

class data_migrator.contrib.dutch.PhoneAnonymizor[source]

PhoneAnonymizor generates a random dutch phonenumber, like +3142097272

>>> len(PhoneAnonymizor()('020-1234583'))
11
>>> len(PhoneAnonymizor()('06-12345678'))
11
class data_migrator.contrib.dutch.ZipCodeAnonymizor[source]

ZipCodeAnonymizor generates a random dutch zipcode, like ‘4897 LD’

>>> len(ZipCodeAnonymizor()('1234 aa'))
7
data_migrator.contrib.dutch.clean_phone(v)[source]

Cleans phone numbers to dutch clean format

clean_phone clean phone numbers, replaces all characters and spaces adds dutch country code (+31) if no country code is provide

>>> clean_phone('00 31 6 - 20 20 20 20')
'+31620202020'
>>> clean_phone('06 20 20 20 20')
'+31620202020'
>>> clean_phone('020 -123 345 6')
'+31201233456'
>>> clean_phone('+440.203.020.23')
'+44020302023'
>>> clean_phone('+440a203a020a23')
'+44020302023'
>>> clean_phone('+440 ada 203.020 // 23')
'+44020302023'
>>> clean_phone('31 (6) - 20 20 20 20')
'+31620202020'
Parameters:v (str) – value to clean
Returns:cleaned phone number
Return type:str
data_migrator.contrib.dutch.clean_zip_code(v)[source]

Cleans a dutch zipcode

>>> clean_zip_code('1234 aa')
'1234AA'
>>> clean_zip_code('1234AB')
'1234AB'
>>> clean_zip_code('1234   Ba')
'1234BA'
>>> clean_zip_code('1 2 3 4 A B')
'1 2 3 4 A B'
>>> clean_zip_code('blabla')
'blabla'
Parameters:v (str) – zipcode to clean
Returns:cleaned zip code
Return type:str