1、(1)逆向APP时,第一个干的事就是抓包了,抓包的结果类似下面:
GET https://aweme.snssdk.com/aweme/v1/commit/item/digg/?aweme_id=6956180208793718055&type=1&channel_id=-1&city=510100&activity=0&os_api=22&device_type=M973Q&ssmix=a&manifest_version_code=150501&dpi=480&uuid=865166023654745&app_name=aweme&version_name=15.5.0&ts=1620296753&cpu_support64=false&app_type=normal&appTheme=light&ac=wifi&host_abi=armeabi-v7a&update_version_code=15509900&channel=xiaomi&_rticket=1620296754623&device_platform=android&iid=2744832902828125&version_code=150500&cdid=7addad62-d097-41da-b852-e0de8b24fe75&openudid=338803367b93a120&device_id=70039083151&resolution=1080*1920&os_version=5.1.1&language=zh&device_brand=Meizu&aid=1128&minor_status=0&mcc_mnc=46000 HTTP/1.1 Accept-Encoding: gzip x-tt-dt: AAA7NE2KX2XRH5ZIBMIIJHUGNWEXQLPT2G5U2VKAPRSBVVGBNOLZGYT36N7VMCWTDFTQBQXBW6FSC2IWBD36QSIH6MSUI2SKDMHF4T27M5MS6TL2XHUCXVFWWDJM2Y3HQZEHXQ42UCB7JUEHURQRH6Y passport-sdk-version: 18 X-Tt-Token: 000e8f473aea647b3c44fd3b9f7482439e02da23d858b1609a77f5edbf3bcc6c774f39175ea27eed09ffb2dee02c173fcc137b53019bb3614db48a84e88879cf2cabc8a810d38b88fb1d9bc7640b02e655a186f8b1e0daf197e60067b35a68adc68ad-1.0.1 sdk-version: 2 X-SS-REQ-TICKET: 1620296754625 Cookie: passport_csrf_token_default=f4442957bc07adf1986f4df139296dd7; install_id=2744832902828125; ttreq=1$184784690dc292ea1ad4e5db65553166a5674c3e; multi_sids=95063141447%3A0e8f473aea647b3c44fd3b9f7482439e; odin_tt=fac32cc300e49d8f22a70f1cd7b92569039d7305b6fdef7f2389cb98e0f9e9493f6c8374ad33dabed24ea41c55e1834e; n_mh=B6WRe0yd-1qIuffF6ZWNO-CSGlW1Q-VhC0E79NrqYTg; sid_guard=0e8f473aea647b3c44fd3b9f7482439e%7C1620296607%7C5184000%7CMon%2C+05-Jul-2021+10%3A23%3A27+GMT; uid_tt=d06498fc7329b7187e209d0e6279e1d7; sid_tt=0e8f473aea647b3c44fd3b9f7482439e; sessionid=0e8f473aea647b3c44fd3b9f7482439e; X-Ladon: gW1IO3ulq0eYymnbYa+7nAu2l116ADdAdmSIA1eB3Cv5BBIo X-Khronos: 1620296754 X-Gorgon: 0404401f40051f42c987ec09aebf8038d629adf2add584933f8f X-Tyhon: s2S+nKZ8jrepKJ2VtCulu7carp/ZLqe2gn24aHA= X-Argus: Uictv+neuH8ZqjOjXzvEIvXCEXYEgUy3dUKWzj08JUtmGeYa4HfNfN8bp7Yga22Jbik2N2dBKezA48YN1E9A1KaBjXi2ixu5cHzAkQU9Tl4f/+a9xWuLYIZ/+PkW/YXUsmRCfszWlcMePPeyNQYEGkb5FssTkIr3EZ87TJ9NLBDJCJGA0i4PICbLnClzdfmBs+57JVi1sU2/MELCr9gO/Nka5eIJsEoGB/CIaL3gPmEbZ0sUl7sxIvKksMwj3f7tYBCriYBKmfeREuiYf1S18c7i Host: aweme.snssdk.com Connection: Keep-Alive User-Agent: okhttp/3.10.0.1
client和server通信靠的全是这类API的方式:要么是GET,要么是POST!后台又是怎么生成这些API接口的了?还是要靠springboot!文章末尾参考1有个工程,把unidbg和springboot结合起来了,用起来很方便,demo如下:
package com.anjia.unidbgserver.web; import com.anjia.unidbgserver.service.TTEncryptServiceWorker; import lombok.SneakyThrows; import lombok.extern.slf4j.Slf4j; import org.springframework.http.MediaType; import org.springframework.web.bind.annotation.RequestMapping; import org.springframework.web.bind.annotation.RequestMethod; import org.springframework.web.bind.annotation.RestController; import javax.annotation.Resource; /** * 控制类 * * @author AnJia * @since 2021-07-26 18:31 */ @Slf4j @RestController @RequestMapping(path = "/api/tt-encrypt", produces = MediaType.APPLICATION_JSON_VALUE) public class TTEncryptController { @Resource(name = "ttEncryptWorker") private TTEncryptServiceWorker ttEncryptServiceWorker; /** * 获取ttEncrypt * <p> * public byte[] ttEncrypt(@RequestParam(required = false) String key1, @RequestBody String body) * // 这是接收一个url参数,名为key1,接收一个post或者put请求的body参数 * key1是选填参数,不写也不报错,值为,body只有在请求方法是POST时才有,GET没有 * * @return 结果 */ @SneakyThrows @RequestMapping(value = "encrypt", method = {RequestMethod.GET, RequestMethod.POST}) public byte[] ttEncrypt(String key,String body1) { String key1 = key; String body = body1; /*String key1 = "key1"; String body = "body";*/ // 演示传参 http://127.0.0.1:9999/api/tt-encrypt/encrypt?key=aaaa&body1=bbbbb byte[] result = ttEncryptServiceWorker.ttEncrypt(key1, body).get(); log.info("入参:key1:{},body:{},result:{}", key1, body, result); return result; } }
类上面加了一个 @RequestMapping(path = "/api/tt-encrypt", produces = MediaType.APPLICATION_JSON_VALUE) 的注释,表明请求的API中,凡是有“api/tt-encrypt”路径的,都在这个类内部的方法中处理;再看类中有个方法叫ttEncrypt的,上面也带了@RequestMapping(value = "encrypt", method = {RequestMethod.GET, RequestMethod.POST})这样的注释,说明这个方法的访问路径是“/api/tt-encrypt/encrypt”,访问的方法可以是GET,也可以是POST!所以如果client想调用server的ttEncrypt方法,windows下完整的路径如为:http://127.0.0.1:9999/api/tt-encrypt/encrypt?key=aaaa&body1=bbbbb ,这样server就会执行ttEncrypt方法,然后把执行结果反回给client,是不是很方便了? 正式如此,spring相关的“脚手架”一跃成为java应用层开发的“一哥”! 在浏览器调用API接口时,能看到的日志如下:
2022-08-16 16:19:59.054 INFO 9280 --- [nio-9999-exec-8] c.a.u.web.TTEncryptController : 入参:key1:aaaa,body:bbbbb,result:[116, 99, 3, 0, 0, 1, -97, -48, -122, 106, -96, -53, -48, 50, 57, 51, -46, -46, -4, -116, 32, -20]
(2)这种API接口的设计,业界最流行的莫过于rest风格了!这种风格的接口有几个要点:
2、(1)大家平时常用的网址都很长,为了方便传输和使用【比如跨设备优化链接、跟踪单个链接以分析受众和活动绩效,以及隐藏关联的原始URL等】,短网址生成服务孕育而生(weibo这类对用户发帖长度有限制的站点尤为常见),其实原理也很简单:用户提供long url长网址,有专门的服务用特定的算法把long url转换short url!当用户访问short url时,转换服务会根据short url把long url找到,然后返回给client,并提示client需要301 redirect!过程图示如下:
设计这种长、短网址转换有两个最核心的点:
(2)先来看看转换的算法,业界主要有两种:随机生成和进制转换!
(2.1) 随机转换:从逻辑上讲,long url和short url并无直接的关联关系,怎么匹配完全可以开发人员自己确定,所以最直接、简单、粗暴的方法是随机生成一个字符串就能做为short url返回到client!注意:因为short url是随机生成的,返回给client前肯定要去重,所以理论上讲:short url生成的越多,去重的耗时越长,整个系统的效率就越低!
(2.2)进制转换:上面第一种生成随机数的方法越往后效率越低,核心原因就是short url多了以后去重很耗时,怎么改进这个了? 既然生成的随机数不可控,那改成可控的自增数(auto increment sequential ID)的是不是就能解决去重耗时的这个问题了?显然是可行的!唯一的问题就是怎么把自增数转成0-9、a-z、A-Z组成的url了? 本质上是个62进制转换的问题了!计算过程说明如下:
char
map[] =
"abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"
.toCharArray();
所以12345转成short url的结果就是dnh(当然不足6位的可以用0补齐)!代码如下:
// Java program to generate short url from integer id and // integer id back from short url. import java.util.*; import java.lang.*; import java.io.*; class GFG { // Function to generate a short url from integer ID static String idToShortURL(int n) { // Map to store 62 possible characters char map[] = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789".toCharArray(); StringBuffer shorturl = new StringBuffer(); // Convert given integer id to a base 62 number while (n > 0) { // use above map to store actual character // in short url shorturl.append(map[n % 62]); n = n / 62; } // Reverse shortURL to complete base conversion return shorturl.reverse().toString(); } // Function to get integer ID back from a short url static int shortURLtoID(String shortURL) { int id = 0; // initialize result // A simple base conversion logic for (int i = 0; i < shortURL.length(); i++) { if ('a' <= shortURL.charAt(i) && shortURL.charAt(i) <= 'z') id = id * 62 + shortURL.charAt(i) - 'a'; if ('A' <= shortURL.charAt(i) && shortURL.charAt(i) <= 'Z') id = id * 62 + shortURL.charAt(i) - 'A' + 26; if ('0' <= shortURL.charAt(i) && shortURL.charAt(i) <= '9') id = id * 62 + shortURL.charAt(i) - '0' + 52; } return id; } // Driver Code public static void main (String[] args) throws IOException { int n = 12345; String shorturl = idToShortURL(n); System.out.println("Generated short url is " + shorturl); System.out.println("Id from url is " + shortURLtoID(shorturl)); } } // This code is contributed by shubham96301
(3)因为要建立long url和short url的映射关系(方便查询),并且要持久化存储在磁盘,所以肯定是要使用数据库的!sql和NoSql怎么选了?挨个分析一下需求:
综上所述:存放的数据并不多,并且QPS也不高,配置稍微高点的单机服务器(cpu>=24core、内存>=256G、SSD>=12T等)都能满足需求!为了提高QPS,还可以在内存用hashMap缓存mapping关系!假设除去操作系统等必要的开销,还有200G的memory可用,理论上可在memory cache大约970million的mapping关系!由于内存的速度比硬盘快了好多数量级,TPS也能提升好多倍!
如果选用sql型数据库,比如mysql,那么表单schema的设计如下:
当然也可以是short url和long url:
因为sql型数据库可以根据不同的列建索引,所以上面两种schema二选一即可!但是如果用Nosql数据库,由于无法对不同的列建索引,所以可能需要两张表,这里以 Cassandra 为例子:
参考:
1、https://github.com/anjia0532/unidbg-boot-server unidbg+springboot多线程
2、https://cloud.tencent.com/developer/article/1872330 url短链接设计
3、https://www.geeksforgeeks.org/how-to-design-a-tiny-url-or-url-shortener/ 转换代码