Skip to content

Latest commit

 

History

History
119 lines (105 loc) · 26.5 KB

mask.md

File metadata and controls

119 lines (105 loc) · 26.5 KB
Raw

Data Mask

Mask Functions

Function Name Description Arguments Config Example Old Data Masked Data
LaplaceDPFloat64 differential privacy masking based on laplace 1: l0sensitivity, 2: lInfSensitivity, 3: epsilon, 4: delta LaplaceDPFloat64,100,1,1,0 3200.53 3074.3990583359264
LaplaceDPInt64 differential privacy masking based on laplace 1: l0sensitivity, 2: lInfSensitivity, 3: epsilon, 4: delta LaplaceDPInt64,100,1,1,0.5 3200 3198
FF1 format-preserving encryption, ff1 algorithm 1: radix(default 10, max size 62, min size 2, [0-9a-zA-Z]), 2: key, 3: tweak FF1,10 123 009
FF3 format-preserving encryption, ff3 algorithm 1: radix(default 10, max size 62, min size 2, [0-9a-zA-Z]), 2: key, 3: tweak FF3,10 123 080
Base64 base64 encryption no argument Base64 abc YWJj
DES des encryption 1: key, 2: iv DES hello world jZLr1ir1An0IQc30XbLL3A==
AES aes encryption 1: key, 2: iv AES hello world F5QMumnZOlCchKi2nu99rA==
TDEA tdea encryption 1: key, 2: iv TDEA hello world F0HMxhk+uKKBSlR1IAyt+Q==
AESCTR aesctr encryption 1: key, 2: iv AESCTR hello world 0URdmXIvWjcZS3U=
RSA rsa encryption RSA hello world ... ...
ECC ecc encryption ECC hello world ... ...
Fake generate different type of fake data 1: type(name, address, license-plate, email, ssn, birthday, creditcard, url, number, uuid, ip, ipv6) Fake,Name Any string Auto generate fake string
CRC32 crc32 hash no argument CRC32 abc 352441c2
MD5 md5 hash no argument MD5 abc 900150983cd24fb0d6963f7d28e17f72
SHA1 sha1 hash no argument SHA1 abc a9993e364706816aba3e25717850c26c9cd0d89d
SHA2 sha2 hash no argument SHA2 abc ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad
HMAC hmac hash 1: hash function. e.g., md5, sha1, sha2 HMAC,md5,"pass" hello world 37c4d226765f06daa3ad91a6c33a5d3e
Mangle desensitization with corpus 1: corpus name, 2. secret Mangle,"en_US","secret password" hello world! spars dream!
Phone phone default mask function Phone 13000000123 130*****123
Mail mail default mask function Mail [email protected] z**********@*******m
Username username default mask function Username Dave Li D******
Domain domain default mask function Domain example.com e*********m
CreditCard credit card default mask function CreditCard 6227612145830440 62276121****0440
PersonalID personal id default mask function PersonalID 110223700003697 110223*****3697
USCC china unified social credit code default desensitize method USCC 71797173LM37QP0D4H 717971********0D4H
Age age round Age 46 40
Birthday birthday default desensitize method Birthday 2020-05-34 NNNN-NN-NN
IP IP address desensitize method IP 192.168.0.1 127.0.0.1
Password password default desensitize method Password asfa@12323ssda *********
Salary salary default desensitize method Salary 1350 1000
LicensePlate licensePlate default desensitize method LicensePlate 鄂D71D44 鄂****4
OrganizationCode organizationCode default desensitize method OrganizationCode 100000439 100****439
Shuffle shuffle and keep data type 1. corpus name. e.g., 0-9, Lower, Upper, Chinese Shuffle 1234567890 4802731596
ShuffleRight shuffle right and keep data type 1. index ShuffleRight,4 1234567890 1234731596
ShuffleLeft shuffle left and keep data type 1. index ShuffleLeft,4 1234567890 4802737890
Rot letter substitution with the Nth letter after it in the alphabet 1. radix. e.g., 47, 18, 13, 5, 32768 Rot,5 1234567890 6789012345
Morse morse code no argument Morse def .- -... -.-.
Caesar caesar code 1. index Caesar,3 abc def
Smoke replace every characters with mask 1. replacement Smoke,"*" 123 ***
SmokeLeft replace left n characters with mask 1. index, 2. replacement SmokeLeft,2,"*" 123 **3
ReserveLeft reserve left n characters, smoke right 1. index, 2. replacement ReserveLeft,2,"*" 123 12*
SmokeRight replace right n characters with mask 1. index, 2. replacement SmokeRight,2,"*" 123 1**
ReserveRight reserve right n characters, smoke left 1. index, 2. replacement ReserveRight,2,"*" 123 *23
SmokeMargin replace margin n characters with mask 1. index, 2. replacement SmokeMargin,2,"*" 123456 **34**
ReserveMargin reserve margin n characters, smoke inner 1. index, 2. replacement ReserveMargin,2,"*" 123456 12**56
SmokeOuter replace left, right outer n characters with mask 1. left index, 2. right index, 3. replacement SmokeOuter,2,1,"*" 123456 **345*
ReserveOuter reserve left, right outer n characters, smoke inner 1. left index, 2. right index, 3. replacement ReserveOuter,2,1,"*" 123456 12***6
SmokeInner replace left, right inner n characters with mask 1. left index, 2. right index, 3. replacement SmokeInner,2,1,"*" 123456 12***6
ReserveInner reserve left, right inner n characters, smoke outer 1. left index, 2. right index, 3. replacement ReserveInner,2,1,"*" 123456 **345*
SmokeCharLeft mask left before specify char 1. index, 2. replacement SmokeCharLeft,"@","*" [email protected] ***********@example.com
SmokeCharRight mask right after specify char 1. index, 2. replacement SmokeCharRight,"@","*" [email protected] zhangsan123@***********
Replace string replace 1. old string, 2. replacement, 3. n(-1 for all) Replace,2,"*",-1,"*" 123 1*3
RegexpReplace string regexp replace 1. regexp, 2. replacement RegexpReplace,"[bc]","*" abcdef a**def
RegexpRandomReplace regexp random data replace 1. regexp, 2. min, 3. max RegexpRandomReplace,"^1[3-9][\d]{9}$",5,10 13782430405 13614299600
Reverse reserve string no argument Reverse abc cba
ToUpper string to upper no argument ToUpper abc ABC
ToLower string to lower no argument ToLower ABC abc
Const replace string with const 1. replacement Const,"MASKED" abc MASKED
Number2Const replace all number to 9 1. replacement, default 9 Number2Const (+086)130-1234-123 (+999)999-9999-999
Char2Const replace [a-zA-Z] to N 1. replacement, default N Char2Const abc-def NNN-NNN
NumberFloor number round, replace lower n num with 0 1. index NumberFloor,2 123.23 100
DateRound date round 1. data format, 2. accuracy: second, minute, hour(default), day, month, year DateRound,hour 2021-07-23 17:26:45 2021-07-23 17:00:00
Dateformat convert date format 1.old date format 2.new date format dateformat,"YYYY-MM-DD HH:mm:ss", "YYYY/MM/DD HH-mm-ss" 2021-07-23 17:26:45 2021/07/23 17-26-45
LoopMoveLeft string loop move to left 1. index LoopMoveLeft,3 abcdefg defgabc
LoopMoveRight string loop move to right 1. index LoopMoveRight,3 abcdefg efgabcd
TruncateLeft truncate left n characters 1. index TruncateLeft,2 abcdef ef
TruncateRight truncate right n characters 1. index TruncateRight,2 abcdef ab
Abbreviate english words abbreviate Abbreviate strategy-limited stg-ltd
Initialism english words initialism Initialism hello world hw
Numeronym a number-based word Numeronym internationalization i18n
SM2 SM2 algorithm SM2 hello world ... ...
SM3 SM3 algorithm SM3 hello world ... ...
SM4 SM4 algorithm SM4 hello world b9b1742de155fe5720c0b8b1b95e3134

Rand Seed

--rand-seed flag is used for generating random keys or values. Its default value is the current Unix nano timestamp.

Corpus

Generate a list of words using your system dictionary.

aspell -d en dump master | aspell -l en expand > mask/corpus/mangle.en_US

Generate maskFuncs

~ $ make mask-typo

Mask JSON data

If sensitive data contains in TEXT or JSON data type, use json mask function mark the column first, and config json keys mask rules in mask config file as columns mask rules.

$ cat mask.csv
j,json
password,smoke
firstname,ReserveLeft,1
lastname,ReserveLeft,1

$ cat query.sql
select '{"Userinfo": {"Firstname": "Bruce", "Lastname": "John", "Password": "123456"} }' as j from dual

$ d18n --defaults-extra-file test/my.cnf --mask mask.csv --query query.sql
+--------------------------------------------------------------------------+
|                                    J                                     |
+--------------------------------------------------------------------------+
| {"Userinfo":{"Firstname":"B****","Lastname":"J***","Password":"******"}} |
+--------------------------------------------------------------------------+