HomeBlogThoughtsProject

Thoughts

This is a mirror of my Telegram channel, where I share some of my findings and thoughts on technology.
https://github.com/microsoft/typescript-go 微软选择使用 go 而非 rust 重写 tsc，这下前端圈又热闹了。 大多数的讨论都集中在为什么不选用 rust，这显然是最极致的选项。 从我的视角来看，相比当前的 tsc，使用 go 之类的编译语言重写后，可以显著的降低冷启动的时间，这对于 tsc 这类非持续运行的进程，这非常有帮助（ts 似乎没有走 lsp 这条路线)。 当然使用 rust 可能在冷启动、内存利用等各方面比 go 更优秀，但代价就是项目维护成本陡增，roi 不够突出。而大家诟病的 go 的 gc 机制影响性能，但可能没等 gc 运行，tsc 进程已经执行结束了。 当下重写前端工具链，大家倾向于无脑选 rust，似乎不选 rust 这个项目就没人关注，真是一种技术圈的迷思。既然老大哥做出不一样的选择，未来大家选择的时候，应该得衡量下是否 go 也是合适的选项。
March 11, 2025
看到 brew 终于有 Trae 的 cask 了，安装一个试试看。之前体验过AI生成生成前端的工具，但是从来没有尝试过生成一个正儿八经的小工具。今天试试看用 Trae builder 模式“复刻”了下我的一年前的写的小工具，可以用来计算免息分期的实际价值。 老版本 https://0apr.sorcererxw.com/ 新版本 https://0apr-ai.sorcererxw.com/ 整个开发过程全是 AI （Claude-3.5-sonnet），没有改过一行代码，看着效果还不错，甚至胜过我之前的版本 同时也供君品鉴AI写的代码：https://github.com/sorcererxw/0apr-ai
February 28, 2025
有了推理模型之后，工作中一些复杂算法代码，目前都是交给 LLM 去生成。深刻感受到，之前鼓吹了很久 TDD，在 AI 编码之后变得非常容易： 1. 根据描述，让 AI 生成一组事无巨细的 test cases。 2. 然后让 AI 生成具体代码逻辑就行，根本不需要 review 内部逻辑，单测出错了就让它一直改到对为止，必要的时候给点提示。 颇有产品经理使唤 QA 和 RD 的感觉。
February 26, 2025
手上的一个需求，需要保存一个巨型结构体到数据库当中，得选用一个序列化方式进行存储，需要尽可能保证数据体积小。 这个结构体当中最多的就是大量的 int64 数字，很显然这种情况下使用 protobuf 相比 json 有优势，如果再搭配上压缩算法就更极致了。 一顿折腾测试了下，发现 pb 数据体积确实远小于 json 序列化，但是两者经过 zstd 压缩之后，体积居然相差无几。 那不如直接 json + zstd 就好了，还省得定义 schema。
July 24, 2024
最近在写一个 TUI 的程序，深度使用了一下风很大的 bubbletea，锐评一下： 1. 易于构建大型复杂界面 将界面上每一个元素抽象为 model，拆分逻辑执行函数 Update 和视图渲染函数 View，在组件 Update 当中可以消费全局各种消息更新自身状态，View 当中专注渲染画面，很像 MVVM 架构。通过这种方式可以很容易创造出事件驱动的 UI 架构，对于构建复杂的交互程序很有帮助。 2. 存在界面刷新闪烁问题 可能因为封装比较浅，每一个组件渲染是输出字符串，框架视角下每次渲染是获取整个页面的字符串，很难做到根据变更区域按需要渲染，每次发生元素变更可能导致整行或者整页刷新，页面会闪烁一下，体验比较差。 对比我常用的 TUI 工具，在相同的terminal+shell下，zellij、lazygit、bottom 等，它们并有类似的情况。
July 1, 2024
https://planetscale.com/docs/concepts/hobby-plan-deprecation-faq PlanetScale 居然开始全面收费了，原来的免费实例会自动休眠。我有几个定时任务还在依赖 PlanetScale 的 MySQL，刚发现已经好些天没有成功运行了。研究了半天，我才发现是 Planetscale 的 DB 挂了，太无语了。 https://planetscale.com/blog/planetscale-forever 估计对于 PS 来说，云数据库 SaaS 业务增长基本到头了，再深入得横向扩展很多业务。倒不如砍掉营销和销售，保持一个收支平衡的状态，专注服务好当前的企业客户就好。
April 16, 2024
有一台闲置的 mac，打算用来当作家里的网络代理服务器。有几个方案： - （现状）在软路由里面安装 Clash，指定部分设备流量走 Clash，实现透明代理 - 缺点：手机电脑日常使用 Surge，只有路由器使用 Clash，需要额外维护一份 Clash 配置，麻烦 - 使用 Mac Surge 接管 DHCP，让流量走 Surge 过，实现透明代理 - 缺点：稳定性差，一旦 Mac 挂了，整个网络就瘫痪了 - 特定设备主动配置代理服务器为 Mac Surge - 缺点：不是透明代理，可能部分设备无法设置 Proxy - 在方案一的基础上，让 Clash 直接将所用流量通过 socks5 转发到 Mac Surge，Clash 当中只需要配置一条固定的 socks5 策略 目前来看，最后一个方案既能实现透明代理，又不会影响稳定性，还能实现全设备使用 Surge。
意识到这个模式，使用 iPhone 或者 AppleTV 替换 Mac 也可以实现。因为 AppleTV 不方便插网线，故选择了一台闲置的 iPhone12 搭配 lighting 以太网适配器进行测试。碰到两个问题： lighting 接口理论带宽有 480Mbps，因为在一个接口上下载并转发，实际吞吐还得打 5 折。实际测试下载，最高下载速率只有 10MB/s，实在不够看。 有一个workaround，让iPhone同时接入Wifi，因为有线网卡和无线网卡是不一样的MAC地址，可以分配不同的IP，软路由侧对这两个 IP 负载均衡，那么就可以实现低延迟需求走有线，高带宽需求走无线。 因为经过一层转发，Surge无法识别请求来源设备了，所有来源地址都是软路由。之前使用 SRC-IP 来识别 XBOX 让游戏通通走低延迟的机场，现在没法实现了。
April 7, 2024
有一台闲置的 mac，打算用来当作家里的网络代理服务器。有几个方案： - （现状）在软路由里面安装 Clash，指定部分设备流量走 Clash，实现透明代理 - 缺点：手机电脑日常使用 Surge，只有路由器使用 Clash，需要额外维护一份 Clash 配置，麻烦 - 使用 Mac Surge 接管 DHCP，让流量走 Surge 过，实现透明代理 - 缺点：稳定性差，一旦 Mac 挂了，整个网络就瘫痪了 - 特定设备主动配置代理服务器为 Mac Surge - 缺点：不是透明代理，可能部分设备无法设置 Proxy - 在方案一的基础上，让 Clash 直接将所用流量通过 socks5 转发到 Mac Surge，Clash 当中只需要配置一条固定的 socks5 策略 目前来看，最后一个方案既能实现透明代理，又不会影响稳定性，还能实现全设备使用 Surge。
April 2, 2024
https://github.com/valkey-io/valkey/issues/18 看到 Linux 基金会分叉了 Redis 建立了 Valkey，想起前几天另外一个 Redis 分叉 Redict，好奇两个分叉是什么关系，特地去搜了下。 看到这个 issue，Redict 维护者疯狂推销他们的许可证以及 FOSS 协作工具，尝试合并两个开发者社区。而 Valkey 维护者显然更加务实， 不接受更换代码托管和证书。 管窥开源社区的政治生态，挺有意思的。
March 29, 2024
最近退订了Github两项付费订阅：GitHub Pro($4/m) 和 GitHub Copilot（$10/m) GitHub Pro： 一开始订阅诉求是更多的 Codespaces 的单位时间额度，临时开发一些东西可以直接云端编码。 但是用着用着发现机器配置很不灵活，2C8G 和4C16G 的机器只有 32GB 磁盘空间，大一点的仓库下载一下依赖就占满了并禁止磁盘写入了。 如果想要更大的空间，8C32GB 的机器起步开始提供 64GB 的磁盘空间，这样一来 CPU 单位时间配额马上就用完了。 目前使用 Gitpod 了，无法选择 region，网络连接不如 Codespaces，但是配置足够用，实际开发体验可以接受。 GitHub Copilot： 大模型代码补全也发展好几年了，在大多数场景下，各类竞品完全能够提供不输 Copilot 的体验。 促成我寻找替代品的关键是，在远程开发模式下，Copilot 插件总是无法稳定使用 vscode 或者 goland 的 proxy 配置，导致国内的开发机无法正常补全代码。 目前转向使用 Codeium 了，提示速度、质量都非常好，还在代码内行内提供了一些预先配置好的 prompts 用来快速修复错误/重构/注释，功能上更加丰富。
March 18, 2024
https://blog.jetbrains.com/go/2024/01/18/goland-2024-1-eap-program-and-roadmap/ Goland 2024.1 Beta 更新了，日志上明确提升了性能和 remote dev 稳定性。 今天使用 Gateway 连接了开发机新版的 Goland，整体流畅度相比 2023.3 确实明显提升。之前版本在切换文件或者打开 UI 组件的时候经常会有明显的掉帧，在新版上少了很多类似的情况。 确实可堪一用了，终于不需要使用 VSCode Remote 了。 不过性能开销似乎没啥提升，32c64g 的开发机日常是 20% CPU load + 90% 的内存占用。。。
March 12, 2024
I internationalized the content of my blog using GPT4, and after half a month, the number of Google search impressions has increased significantly. Since all the content is rich text, or even a superset of Markdown, in order to translate and retain the format, it is necessary to define some DSLs. At this time, we have to rely on GPT4's strong logical reasoning ability to ensure the reliability of the output results. However, during the debugging process, repeatedly adjusting the prompts, the cost of GPT4 is too high, and it is a bit unbearable. I'll wait until GPT4 Turbo is released and consider automatically monitoring content changes and automatically updating translated content.
December 10, 2023
Let's take a look at various analytics SaaS. Apart from Google Analytics, almost all similar services are not exactly cheap. Services like Vercel/Cloudflare cost about $20 a month. If it's purely for website data statistics, it doesn't seem to be of great significance, especially when the traffic is not high. The cost of setting up something like umami is less than $2 a month. For me, a bigger pain point is actually the observability of the server side. It's quite troublesome to maintain a set of prometheus/Influxdb + grafana. So I plan to find an Analytics SaaS that natively supports OpenTelemetry, and I will directly connect the data from the front end and back end through OpenTelemetry. The most ideal state is to be able to integrate log/metrics/trace/alert all into one platform. After looking around, either the price is very high, or the functions are not complete, or it is not compatible with OpenTelemetry. Thinking about it, once the data dimensions of this type of data statistics increase, it is easy to see a magnification phenomenon where the traffic of statistical data is several times that of business traffic. I also maintain a statistical service at work, which is a cost monster of the entire system. I can understand why various data statistics SaaS are so expensive.
Finally, I set up an OpenTelemetry system, using Otelcol to collect and report data, Prometheus to store metrics, and Tempo to store traces. Finally, all data is viewed in a unified manner on Grafana, and alarm capabilities can also be implemented in Grafana. Considering that PaaS (Railway) already provides free log storage, I will no longer maintain it myself. I only need to ensure that the log contains the trace ID to facilitate association and traceability. All components are deployed with only one instance of Docker. Except for Prometheus, which takes up more memory, the overall overhead is not large. Compared with purchasing third-party SaaS services, it is more cost-effective.
December 10, 2023
https://www.recursive.design/ I really love Recursive, this variable font. As a UI font, my favorite feature is that, under the fixed font-family (sans/mono), adjusting the font weight does not affect the font width. When doing front-end development, if we adjust the font-weight of the corresponding element when we hover, it often causes jitter, and we need to use shadow to replace font-weight to achieve the same effect. If we use Recursive, we can elegantly avoid this problem. Because it supports the monospace variant, I am currently also using Recursive as the font for my code editor, which looks very good.
July 18, 2023
A few weeks ago, I added build notification capabilities to my service: after CI/CD completion, a Telegram Bot would notify me. Today, I set this bot to mute because there were often a lot of unread messages piled up, and I was too lazy to read them...... It's actually a strange psychology: - If no notifications are sent at all, I have no idea about the deployment status - If only deployment failure notifications are sent, not receiving a message doesn't guarantee success, and the CI/CD process might not even have been triggered - If all notifications are sent, the proportion of valid information is too low, and I'm too lazy to read it
July 14, 2023
https://blog.cloudflare.com/cloudflare-snippets-alpha/ Interesting, Cloudflare's Snippets, can be placed in front of the API to perform some lightweight business logic, somewhat similar to an API Gateway. However, it's strange why they didn't directly call it middleware or interceptor, which would be more in line with common sense.
June 26, 2023
For SEO purposes, I have synchronized the content of this channel using the Telegram API, which is displayed on my homepage. Subsequently, the content of this channel can be searched through Google. https://sorcererxw.com/thoughts
June 22, 2023
https://buf.build/blog/connect-a-better-grpc The Buf team, a Protobuf management tool, has released an RPC suite called Connect. They listed some problems with gRPC: ⁃ Too complex, hard to debug, and the large codebase is prone to vulnerabilities ⁃ Does not use the net/http standard library ⁃ Does not support browsers Connect is an optimization of gRPC, with some features: ⁃ Simplified code, including more readable generated code ⁃ Uses the net/http standard library, better compatibility ⁃ Supports three protocols: gRPC/gRPC-Web/Connect ⁃ Only supports HTTP POST method, supports both HTTP/1.1 and HTTP/2, and both pb and json data formats ⁃ Supports the full gRPC protocol, including server reflection and health checks. Compared to Twitch's twirp, Connect is still compatible with the gRPC protocol, while twirp is more like a JSON-RPC based on the Protobuf generator. It seems that Connect is indeed "A better gRPC", which can cater to high-performance scenarios and also do fallback for restricted environments (browsers/debugging).
Suddenly, I realized that using an Interface Definition Language (IDL) like Protocol Buffers (protobuf) or Thrift often allows for the modification of field names, as long as the field types and numbers are not changed, ensuring the consistency of the data structure and memory structure, and thus avoiding conflicts between the client and the server. However, if you use a framework like grpc-gateway or Connect, based on protobuf, to interact with the frontend via JSON rather than binary data, it's a different story. In this case, you can freely change the numbers, but you absolutely cannot change the field names. If you want to be compatible with both JSON and binary data (native grpc) transmission, then it means that a field on this IDL cannot be changed. Essentially, this is due to the two modes using different field indexing methods.
June 17, 2023
【New Product Introduction】OpenObserve is a one-stop Logs, Metrics, Traces observability product developed in Rust. Last year, the team developed a lightweight alternative to ElasticSearch, zincsearch, which quickly gained over 15,000 stars and received $4.6 million in seed funding. They later shifted their focus to a new observability product, ZincObserve, which has recently been renamed to OpenObserve. https://openobserve.ai/
June 12, 2023
Recently, I upgraded my self-hosted umami from 1.x to 2.x. According to the documentation, I successfully upgraded the table structure and data in postgres, but after the 2.x instance was up, it could not read the data in the database. Looking at the logs, it seems that prisma had an error processing the data. Helplessly, I tried to clear the old data, the original error was gone, but the data reported on the web page was not correctly consumed and written. Tired, and too lazy to roll back to 1.x, I don't really want to continue using umami.
Let's take a look at various analytics SaaS. Apart from Google Analytics, almost all similar services are not exactly cheap. Services like Vercel/Cloudflare cost about $20 a month. If it's purely for website data statistics, it doesn't seem to be of great significance, especially when the traffic is not high. The cost of setting up something like umami is less than $2 a month. For me, a bigger pain point is actually the observability of the server side. It's quite troublesome to maintain a set of prometheus/Influxdb + grafana. So I plan to find an Analytics SaaS that natively supports OpenTelemetry, and I will directly connect the data from the front end and back end through OpenTelemetry. The most ideal state is to be able to integrate log/metrics/trace/alert all into one platform. After looking around, either the price is very high, or the functions are not complete, or it is not compatible with OpenTelemetry. Thinking about it, once the data dimensions of this type of data statistics increase, it is easy to see a magnification phenomenon where the traffic of statistical data is several times that of business traffic. I also maintain a statistical service at work, which is a cost monster of the entire system. I can understand why various data statistics SaaS are so expensive.
June 9, 2023
Recently, I upgraded my self-hosted umami from 1.x to 2.x. According to the documentation, I successfully upgraded the table structure and data in postgres, but after the 2.x instance was up, it could not read the data in the database. Looking at the logs, it seems that prisma had an error processing the data. Helplessly, I tried to clear the old data, the original error was gone, but the data reported on the web page was not correctly consumed and written. Tired, and too lazy to roll back to 1.x, I don't really want to continue using umami.
June 9, 2023
Recently at work, I had a requirement to process a batch of data, which involved various types of analysis and statistical calculations. The volume of the original data was extremely large, with a Cartesian product of approximately 10^15 data entries to be processed every hour. Because the scale of the data was simply too large, it was impossible to perform real-time statistics online, so we had to calculate it offline slowly. I set up a Spark scheduling task to run every hour, processing the data from the previous hour. Each calculation required thousands of cores and several terabytes of memory, and still took dozens of minutes to run. As a backend developer, I've always been implementing various real-time calculations, meticulously calculating the cost of each computation, and rarely dealing with offline data analysis. I've never seen a scenario like this before, where you throw a SQL statement in, and it just runs with brute force. It's quite shocking.
June 7, 2023
I tinkered with the newly added Ponte networking capability of Surge5 over the weekend. Compared to Tailscale, which I have been using, it has a clear advantage in that it does not require setting up a DERP relay server on your own, and can directly use the airport as a relay server. I used the Hong Kong node as a relay, and the latency when accessing my home Mac from the office is consistently below 100ms. Because airports are often unrestricted in speed, using VNC for remote desktop does not require compressing the image quality. Access to home network services from outside can be achieved through some simple rules: HOME = select, DIRECT, DEVICE:mymac IP-CIDR,192.168.50.0/24,HOME Conversely, you can also directly access the company's internal network at home through the company's Mac as a springboard, which is very convenient.
In the morning, I tried to access the company's intranet at home using Surge as a springboard, but found that I couldn't access it in rule mode. I found that all requests directly used the fallback rule of Final, and accessed the company's intranet through the airport, which is definitely not accessible. Even DNS can't succeed (need to use intranet DNS). After studying for a long time, I found that it was because I added an IP-CIDR rule before the intranet rule. The overall rules are as follows: IP-CIDR,10.0.0.0/8,PROXY DOMAIN-SUFFIX,company.org,COMPANY FINAL,PROXY,dns-failed According to the Surge documentation: When making rule judgments, Surge tries to match each rule from top to bottom. If it encounters an IP type rule, Surge will perform DNS resolution before matching. 1. When dealing with intranet requests, I first encountered the IP-CIDR rule and started DNS query for the domain name on the spot, but the intranet DNS locally obviously couldn't access, and the final resolution failed. 2. According to the dns-failed rule, the FINAL's PROXY strategy was used, and the airport was chosen as the fallback. Finally, after moving the IP-CIDR rule to the bottom, the problem was solved. Honestly, this setting is quite tricky...
June 7, 2023
Macs (especially those with Intel processors) often encounter a situation where the kernal_task maxes out the CPU after waking up. It's speculated that this is due to the system having to read data cached on the disk back into memory, which can cause significant lag. https://discussions.apple.com/thread/5497235 I've tried various solutions, but none have had a noticeable effect. Now I've found a pretty good method, which is to open a terminal and execute caffeinate -s. As long as I don't terminate this process, the system will not go into sleep mode at all, thus avoiding the wake-up issue.
June 2, 2023
I previously used mackup to back up various configuration files on my Mac to iCloud. Today, I found that the mackup directory in iCloud was mistakenly deleted, and I couldn't find it even with file recovery on iCloud... Then iCloud silently synced files and deleted the local version as well. All kinds of configurations were soft-linked to the iCloud directory, resulting in the loss of all local configurations. Now, looking at the only one still open zsh that loaded the previous .zshrc, I'm hesitating whether to give up recovery and reconfigure a .zshrc. Years of accumulated configurations are all gone 😩.
Eventually, I returned to simplicity, opening a repository on Github to store various configuration files, and manually creating symbolic links with ln -s after cloning to the local. For these types of configuration files that don't need to be changed every day, there is indeed no need to use real-time synchronization with cloud drives like icloud. Here I share the rewritten .zshrc, which I use in my multiple macos/linux environments. It is automatically synchronized to Gist from my configuration repository using Github Action, https://gist.github.com/sorcererxw/238f7068c18ba148337f32f9a08d0dbd
May 24, 2023
I spent all night messing with jetbrains gateway last night, thinking about using goland for remote development. The overall experience is terrible. Various functions and plugins need to distinguish between Host and Client, some plugins need to be installed on both sides to be used, and the mental burden is high; many plugins are designed for client mode, and they can't be used at all in this front-end and back-end separation mode. As for performance, it's acceptable. Opening a golang monorepo on an 8c16g development machine, the CPU will be maxed out when indexing, and it will use about 2c8g during off-peak periods. All in all, I'd rather use neovim for remote development, at least the experience is native.
May 24, 2023
I tinkered with the newly added Ponte networking capability of Surge5 over the weekend. Compared to Tailscale, which I have been using, it has a clear advantage in that it does not require setting up a DERP relay server on your own, and can directly use the airport as a relay server. I used the Hong Kong node as a relay, and the latency when accessing my home Mac from the office is consistently below 100ms. Because airports are often unrestricted in speed, using VNC for remote desktop does not require compressing the image quality. Access to home network services from outside can be achieved through some simple rules: HOME = select, DIRECT, DEVICE:mymac IP-CIDR,192.168.50.0/24,HOME Conversely, you can also directly access the company's internal network at home through the company's Mac as a springboard, which is very convenient.
May 8, 2023
I previously used mackup to back up various configuration files on my Mac to iCloud. Today, I found that the mackup directory in iCloud was mistakenly deleted, and I couldn't find it even with file recovery on iCloud... Then iCloud silently synced files and deleted the local version as well. All kinds of configurations were soft-linked to the iCloud directory, resulting in the loss of all local configurations. Now, looking at the only one still open zsh that loaded the previous .zshrc, I'm hesitating whether to give up recovery and reconfigure a .zshrc. Years of accumulated configurations are all gone 😩.
April 28, 2023
https://buf.build/blog/connect-a-better-grpc The Buf team, a Protobuf management tool, has released an RPC suite called Connect. They listed some problems with gRPC: ⁃ Too complex, hard to debug, and the large codebase is prone to vulnerabilities ⁃ Does not use the net/http standard library ⁃ Does not support browsers Connect is an optimization of gRPC, with some features: ⁃ Simplified code, including more readable generated code ⁃ Uses the net/http standard library, better compatibility ⁃ Supports three protocols: gRPC/gRPC-Web/Connect ⁃ Only supports HTTP POST method, supports both HTTP/1.1 and HTTP/2, and both pb and json data formats ⁃ Supports the full gRPC protocol, including server reflection and health checks. Compared to Twitch's twirp, Connect is still compatible with the gRPC protocol, while twirp is more like a JSON-RPC based on the Protobuf generator. It seems that Connect is indeed "A better gRPC", which can cater to high-performance scenarios and also do fallback for restricted environments (browsers/debugging).
The connect server now supports HTTP GET. You just need to set the idempotency level to no side effects, and it will automatically configure the GET router for the corresponding method. service ElizaService { rpc Say(stream SayRequest) returns (SayResponse) { option idempotency_level = NO_SIDE_EFFECTS; } } idempotency_level is a built-in method attribute in protobuf, divided into unknown/idempotent/no_side_effects, the latter two both mean idempotent. In the RPC scenario, as long as it is idempotent, it means that the client can safely retry the RPC call. Whether there are side effects is not strongly instructive. However, when it comes to the http restful interface, GET often equals no side effects, so it is quite reasonable and clever to judge whether to support HTTP GET based on this. https://github.com/bufbuild/connect-go/releases/tag/v1.7.0
April 22, 2023
https://matt-rickard.com/reflections-on-10-000-hours-of-devops
April 3, 2023
In the latest Next.js13 appdir documentation, Vercel introduces the Comments component. You only need to log in to Vercel to comment anywhere on the page. It's very similar to Figma's comment system, except that the canvas is replaced with a webpage. Vercel also developed this capability for Vercel users. All Preview Deployments (non-main branches) can be directly enabled. By default, only members of the organization to which the current Deployment belongs can comment, but permissions can be opened to any user. Once a comment is made, Vercel will sync the comment to the corresponding branch's PR, which can naturally integrate into the development process. Wouldn't it be interesting to create a Disqus service in this mode?
https://cord.com/ Indeed, there is a canvas comment SaaS.
March 27, 2023
https://www.bitestring.com/posts/2023-03-19-web-fingerprinting-is-worse-than-I-thought.html TIL, clearing website data or using incognito mode cannot prevent browser fingerprinting.
March 22, 2023
Recently, I need to implement a super high QPS metric statistics function on Redis. Considering retry idempotency, I can't directly use INCR for accumulation. So I thought about using the Set structure for storage and SCARD to check the total amount. This involves the problem of excessive storage space. Then I went through all sorts of trouble, wrote a bunch of code logic to minimize storage usage, and even switched Redis to a disk-based large-capacity storage similar to Redis. Just now when I was discussing with ChatGPT, it pointed out that I could use HyperLogLog. I was surprised 🙈. Technology needs to be used and updated frequently, otherwise it's too easy to forget without review.
March 16, 2023
I found that my various backend service deployments were a bit messy, with deployments on PaaS such as Vercel, Railway, Digital Ocean, etc., corresponding to different wool usage scenarios. However, I found that it was very troublesome for services to call each other, and the code needed to care about the specific location of the target service, which was not small in terms of maintenance cost and mental burden. So I wrote a "service discovery" mechanism. Using a Discovery Service, through the APIs of Vercel, Railway, Digital Ocean, it continuously pulls deployment information from the corresponding platforms and stores it. Then each service will periodically obtain the full list of services from Discovery, and then obtain the corresponding public network address based on the unique identifier of the service. In this way, as long as the service is deployed on PaaS and exposed to the public network, the client can access it directly through the service name.
March 12, 2023
Previously, through DHCP, I directed the gateway of all devices in the house to the soft router (openclash) to achieve circumvention. However, I recently found that the stability of homekit is not very good, and the home hub often goes offline. I thought about it and it's probably because the connection stability of openclash to the proxy server itself is not enough. So, on the router, I adjusted the DHCP configuration through dnsmasq, and only directed the gateway of some devices to the soft router. Considering that homekit itself may also need to circumvent, I left one home hub (homepod) within the wall, and one (apple tv) turned on the circumvention. In this way, the overall stability has improved a lot.
February 27, 2023
https://mastodon.social/@simevidas/109919980697679274 Although it meets expectations, it's still very funny.
February 25, 2023
I recently tried out railway.app, deployed some services, and here are some of my thoughts: - Similar to Heroku, service instances are not destroyed, so there are no cold start issues. - Billing is based on CPU and memory usage + duration. If the monthly consumption is less than $5, no charges are applied. - If you do not enable the paid plan (link a credit card), there is also a $5 allowance per month, but the total online duration of instances is limited to 500 hours (21 days) per month. In other words, if you want to deploy services for a long term, you must enable the paid plan. - You can deploy databases (PG, MySQL, Redis, MongoDB), and also support accessing and operating the database from the front end. However, I haven't seen any database expansion, backup functions, etc. It seems to simply provide a database container instance, which is not suitable for production. - Railway has the concept of Project, where a Project can have multiple environments, deploy multiple services/databases, much like K8S's Namespace, suitable for business isolation. - It automatically recognizes the Dockerfile in the project to build and deploy images, and the build speed is relatively fast. - Like Vercel, it does not support VPC, which means if you make service-to-service calls, services can only access each other through public domain names, which is relatively insecure. However, this feature is already in the WIP state in the roadmap, so it's promising. - It does not retain historical logs, and you need to rely on external log services like logtail for log storage. - It does not support cronjob, and the solution given in the documentation is to start an instance for scheduling... Overall, for individual developers, the Railway experience is okay. But after all, it's a startup company that's only two years old, and the platform features and documentation are not yet fully developed.
A few days ago, when I was checking my credit card bill, I was surprised to find that the MongoDB Atlas managed database had deducted nearly $200 last month. It might be because I deployed some cronjobs last month, which used a large amount of database. However, it's unacceptable that a few small services generate such high costs. I thought I couldn't afford MongoDB Atlas anymore. But I couldn't find a substitute for MongoDB hosting for the time being, so I tried to start a MongoDB hosting instance on Railway. I dumped the old data over and restarted all related services, and the migration was completed in about ten minutes. I just took a look at the Usage statistics of Railway. Based on the average CPU/MEM usage per minute, I estimated the monthly cost, which would not exceed $5 🫠 (much more cost-effective than charging based on the number of reads and writes). However, as mentioned before, Railway does not have a strong guarantee for the availability of the database, nor does it have an explicit data backup solution. I plan to write a script with Github Action to dump the database to S3 on a schedule, and use it temporarily.
January 31, 2023
The Wikipedia enhancement plugin Wikiwand that has been on hiatus for several years has recently updated to 2.0version. In addition to a more modern UI design, it also adds a TLDR mode based on GPT-3, assuming the reader is a child to let AI summarize the entry. After using it for a few weeks, browsing Wikipedia has become a pleasure.
January 31, 2023
#Copilot Black production is really everywhere......
November 24, 2022
The concept of BFF (Backend For Frontend) is promoted to take over all the dirty and tiring work that is solely for the frontend, allowing business services to focus on running logic. However, in practice, the BFF layer is advocated to be as thin as possible. Often, we can see two extremes. Business services define a bunch of RPC interfaces specifically for frontend services, and BFF becomes an API Gateway. Or, the BFF contains very heavy business logic, and business services become CRUD executors.
November 22, 2022
I recently tried out railway.app, deployed some services, and here are some of my thoughts: - Similar to Heroku, service instances are not destroyed, so there are no cold start issues. - Billing is based on CPU and memory usage + duration. If the monthly consumption is less than $5, no charges are applied. - If you do not enable the paid plan (link a credit card), there is also a $5 allowance per month, but the total online duration of instances is limited to 500 hours (21 days) per month. In other words, if you want to deploy services for a long term, you must enable the paid plan. - You can deploy databases (PG, MySQL, Redis, MongoDB), and also support accessing and operating the database from the front end. However, I haven't seen any database expansion, backup functions, etc. It seems to simply provide a database container instance, which is not suitable for production. - Railway has the concept of Project, where a Project can have multiple environments, deploy multiple services/databases, much like K8S's Namespace, suitable for business isolation. - It automatically recognizes the Dockerfile in the project to build and deploy images, and the build speed is relatively fast. - Like Vercel, it does not support VPC, which means if you make service-to-service calls, services can only access each other through public domain names, which is relatively insecure. However, this feature is already in the WIP state in the roadmap, so it's promising. - It does not retain historical logs, and you need to rely on external log services like logtail for log storage. - It does not support cronjob, and the solution given in the documentation is to start an instance for scheduling... Overall, for individual developers, the Railway experience is okay. But after all, it's a startup company that's only two years old, and the platform features and documentation are not yet fully developed.
November 22, 2022
In the latest Next.js13 appdir documentation, Vercel introduces the Comments component. You only need to log in to Vercel to comment anywhere on the page. It's very similar to Figma's comment system, except that the canvas is replaced with a webpage. Vercel also developed this capability for Vercel users. All Preview Deployments (non-main branches) can be directly enabled. By default, only members of the organization to which the current Deployment belongs can comment, but permissions can be opened to any user. Once a comment is made, Vercel will sync the comment to the corresponding branch's PR, which can naturally integrate into the development process. Wouldn't it be interesting to create a Disqus service in this mode?
November 9, 2022
https://github.com/topgrade-rs/topgrade A single command updates packages in various package managers, including homebrew/zsh plugins/pip/npm/docker images, etc. Not updating will make Star Wars fans feel comfortable.
November 8, 2022
Recently, I need to store a batch of complex configuration files in the database, and I need the ability to atomically update the internal fields of the configuration. Initially, I planned to use MySQL, and there were two schemes: 1. Serialize the entire data structure and stuff it into a field in a row. When updating, read the whole thing out in a transaction (lock the row), update it, and write it back. 2. Based on the data structure, model it honestly, various list tables, relation tables, each field is a column... In most cases, you don't need to use transactions to update internal fields separately. The former is easier, but not elegant; the latter is sufficiently dogmatic, but too troublesome. In the end, I switched to using MongoDB. The entire configuration is written to the library as a document, and you can update the subfields atomically as you wish. Although MongoDB updates documents essentially as single-row transactions, the semantic support for partial updates is indeed more elegant than MySQL.
October 26, 2022
https://vercel.com/analytics Vercel has acquired Splitbee, integrating visitor analysis into its own Analytics. There's no need to specifically introduce Google Analytics for personal projects anymore. I've always wanted to try Splitbee, I like its product tone, now it's more convenient.
October 26, 2022
I've been seriously using Github Codespaces to write code for a while, and configured my local development environment with Github Dotfiles. I have to say the experience was very bad, and countless times I resisted the urge to clone the code to my local machine and open it directly with an IDE. On the one hand, I'm not used to developing with VSCode. Even for front-end development, which is a strong point of VSC, I feel it's not as responsive as Goland. I've also tried developing directly on Codespaces via SSH through Goland, but that seemed like a redundant step. On the other hand, the network is not stable enough. Although it's stable most of the time, once the connection is interrupted, you may find that the local changes to the code can't be committed at all. Refresh the page, and the code might be lost. I was constantly anxious about network issues during development. Well, this completely dispelled my thoughts of buying an iPad Pro.
October 13, 2022
https://mp.weixin.qq.com/s/W_9XYkF5Y_-M-aLdB-b_Ng
October 9, 2022
https://makeavideo.studio/ Meta's AI video generation is too cool!
September 30, 2022
I came across this issue unintentionally. When we use Notion as a CMS to build a website, we can't avoid the problem that Notion's S3 image links will expire. Because the images will expire, the statically generated web pages often can't load the images. Even now, if you go to the Railway blog, you can still find that some images can't be loaded. The simplest way to avoid this problem is not to host the images on Notion, but to use external links to insert into Notion, which is a bit troublesome as it requires uploading to the image hosting first. The issue poster chose to give up SSG and use SSR rendering, which can ensure that the image links sent each time are the latest. This will definitely cause the web page to load slowly (you can't use CDN, if you use CDN, you can't ensure the page is up-to-date). My homepage is also built with Notion, but I have done a lot of pre-rendering work on the server side, so SSR is definitely unacceptable. My current solution is to use a cronjob to pull the website's sitemap, use Next.js's revalidate feature to periodically regenerate each page, and ensure that the newly rendered pages are cached by Vercel's CDN. It's a very crude solution, but it seems to work well so far, and the front-end loading speed is very fast. However, I have another unverified method: Since we host images directly on Notion for convenience, we can regularly scan the page through the Notion API, download the images hosted on Notion, upload them to the image hosting, and replace the original images. Although it seems to solve the problem at the source, it doesn't seem to be very convenient.
Recently, I implemented a more elegant solution, which is to point the src of all notion blocks (img/video/...) on the web page to /api/resource/{block-id}. Each time this interface is requested, it will request the Notion API to get the latest S3 link, and finally return the client HTTP 307 redirect to the resource address, using Cache-Control to ensure that it does not need to be reacquired for a period of time. In this way, every time the browser can jump to the unexpired resource link, there is no need for any external cronjob to actively trigger the refresh.
September 30, 2022
https://vercel.com/changelog/improved-monorepo-support-with-increased-projects-per-repository Vercel has increased the number of projects that a single repo can bind from 10 to 60 for pro users, which can now support a medium-sized monorepo.
September 30, 2022
Cloudflare Announces Message Queue Beta Test Starting from Worker, Cloudflare has been continuously developing the necessary infrastructure to help developers build large, reliable applications. When you encounter situations where you don't need immediate results but need to control concurrency, it's almost time for the message queue to come into play. Limitations Number of queues: 10 per account Message size: ≤128KB Number of message retries: ≤100 times Batch submission of messages: ≤100 Batch processing wait time: ≤30 seconds Message throughput: ≤100 per second *Cloudflare Queues integrates with Cloudflare Workers, and currently, sending and receiving messages must use Worker, other API interfaces will be supported in the future. *Some limitations may be adjusted, relaxed, or cancelled in subsequent tests. Pricing Charged by the total number of operations per month; each write, read, or delete of 64 KB of data is counted as one operation, with no bandwidth traffic fees. The first million operations per month are free, and each subsequent million operations cost $0.4. A complete message transfer requires 3 operations: 1 write, 1 read, and 1 delete. The monthly bill estimate formula is: ( Total number of messages - 1000000 ) * 3 / 1000000 * $0.4 *Free trial during the beta test period >>Apply for Beta Test 🗂Documentation Cloudflare Queues Cloudflare Queues: globally distributed queues without the egress fees ##Message Queue Via @Cloudflare_CN
September 29, 2022
A recent realization is that for personal hobby projects side projects, it's best to use one or two simple tools to complete the core functions. Never introduce a bunch of components, tool chains, and external dependencies to achieve some cool capabilities, and let the product take a step for technology. Who knows if it can still run three months later.
September 27, 2022
https://medium.com/vanguards-of-code/lodash-is-dead-long-live-radash-d9d52abf428b radash - A utility function library replacing lodash It seems to have better and more concise code quality than lodash. Since it is written in TypeScript, the type checking is also more complete compared to lodash. In addition, some utility functions based on async-await have been added. It feels good. https://github.com/rayepps/radash
August 26, 2022
https://hnpredictions.github.io/ This site has crawled all the prediction posts on Hacker News. It's a kind of "automatic grave digging" in a sense. Thanks to the high-quality user group of HN, it's quite interesting to see the predictions about technology, politics, and economy made by users a few years ago.
August 15, 2022
https://deephaven.io/blog/2022/08/08/AI-generated-blog-thumbnails/ Quite interesting, using DALL·E to generate images as article illustrations. In the past, inserting an image involved using keywords to search for it in a search engine, and then pairing the image with an alt. Now, it's about directly pairing an alt with an image, and ensuring that the image is unique and consistent in style, which is indeed revolutionary.
August 13, 2022
https://www.jetbrains.com/idea/whatsnew/ The JetBrains suite 2022.2 has been released, with an update to Goland. The new features are not particularly impressive, and the biggest update this time should be the switch of the runtime from jre11 to jre17. Thanks to the use of the macOS Metal API, it does feel a bit smoother.
July 29, 2022
I have a service that reverse proxies the airport's Clash/Surge rules, and includes some custom proxy/rule (such as intranet jump board, development machine, blocking host) in the output. All clients on all devices only need to subscribe to this reverse proxy address to synchronize the same configuration on all devices. For me personally, this is still very useful. I wonder if there would be an audience for such a SaaS service?
July 22, 2022
Some of my services need redis, so I directly used upstash. At first, I thought the daily 10k request quota was more than enough, but I didn't expect it to exceed the limit in minutes once I started using it. In the end, I obediently ran a redis container on my own server, just for caching. It's a bit less stable, but if the data is lost, so be it.
Recently, I noticed a sudden increase in server usage, and after a long investigation, I found that the Redis cache logic was not executed at all. I checked Redis and was surprised to find that Redis was empty. I tried to set a value that would not expire, but it disappeared after a while. I suspect that this is because Redis did not set a password, was directly scanned on the public network, and then the flushall command was executed. Since Redis only contains some computational caches, which are not very important, so I didn't set up authentication. It seems that no matter what, I should not forget to set a password.
July 18, 2022
I came across this issue unintentionally. When we use Notion as a CMS to build a website, we can't avoid the problem that Notion's S3 image links will expire. Because the images will expire, the statically generated web pages often can't load the images. Even now, if you go to the Railway blog, you can still find that some images can't be loaded. The simplest way to avoid this problem is not to host the images on Notion, but to use external links to insert into Notion, which is a bit troublesome as it requires uploading to the image hosting first. The issue poster chose to give up SSG and use SSR rendering, which can ensure that the image links sent each time are the latest. This will definitely cause the web page to load slowly (you can't use CDN, if you use CDN, you can't ensure the page is up-to-date). My homepage is also built with Notion, but I have done a lot of pre-rendering work on the server side, so SSR is definitely unacceptable. My current solution is to use a cronjob to pull the website's sitemap, use Next.js's revalidate feature to periodically regenerate each page, and ensure that the newly rendered pages are cached by Vercel's CDN. It's a very crude solution, but it seems to work well so far, and the front-end loading speed is very fast. However, I have another unverified method: Since we host images directly on Notion for convenience, we can regularly scan the page through the Notion API, download the images hosted on Notion, upload them to the image hosting, and replace the original images. Although it seems to solve the problem at the source, it doesn't seem to be very convenient.
July 16, 2022
Some of my services need redis, so I directly used upstash. At first, I thought the daily 10k request quota was more than enough, but I didn't expect it to exceed the limit in minutes once I started using it. In the end, I obediently ran a redis container on my own server, just for caching. It's a bit less stable, but if the data is lost, so be it.
June 27, 2022
I have once again annually refactored my personal website built on Notion, switching from using Notion's private frontend interface to its open interface, to avoid as much as possible the occurrence of Breaking Changes. The frequent changes in Notion's private interface often require me to adapt again, which is very annoying. In addition, many Block type fields in the Notion interface are incomplete or do not meet the needs of customized frontend rendering, such as: - Some types of Blocks require secondary queries of child nodes - Code blocks need asynchronous rendering highlighting - Multimedia files require additional authentication - Need to asynchronously generate LQIP - Bookmark type does not contain opengraph information - And so on... I gave up using the data structure provided by the Notion SDK directly on the frontend, and instead used Protobuf to customize a set of data structures, and built a BFF service to aggregate data, completing all the work that needs to be done asynchronously at once on the server side. In this way, only need to fetch data during SSG, without additional calculations on the end, can render the page, whether for SEO or performance can bring improvement. (I am the one who loves to optimize in advance)
June 26, 2022
https://developer.chrome.com/blog/auto-dark-theme/ Turns out Chrome has already built-in an automatic night mode, which is expected to be available to general users in the near future. I tested it, and the overall effect is good, just a bit inferior to Dark Reader. This change is likely to free many designers and developers from the burden of maintaining Dark Mode, only needing to customize dark color schemes for some elements, and leave the rest to the algorithm. For the front-end pages of my personal projects, I also don't plan to spend effort on customizing Palette for Dark Mode anymore (Programming for the future 😁).
June 20, 2022
When using Encore, Google login gave me a prompt telling me that the current email is being used by another account, and I can choose to merge or open another account. It reminded me that I had registered with Github for Encore a year ago. In my opinion, the experience is very good. Every time I log in to a service, if the website provides multiple third-party login methods, such as Google / Github / Twitter, etc., I often get very confused, forgetting which method I used before, and worry that choosing the wrong one will create a useless account. Encore's approach avoids this situation, but it doesn't stop you from opening another account. Technically, the implementation is not difficult to guess: store the user's open id (unique index) on different platforms and the corresponding email (non-unique index) in the user table, and retrieve and compare the user's email when registering. When I integrate third-party login in my personal projects, in order to simplify permissions (in the third-party login process, obtaining user email often requires additional permissions) and database table design, I often only store the open id, but when expanding other login methods in the future, I lose the possibility of account aggregation.
June 17, 2022
https://github.blog/2022-06-14-accelerating-github-theme-creation-with-color-tooling/ Github has released a color system design tool called Primer Prism, which allows for bulk adjustments of HSL on the default color palette to change the entire color scheme. Quite interesting.
June 17, 2022
https://xargin.com/you-should-avoid-orm-and-sql-builder/
June 9, 2022
I discovered a gem on Product Hunt, jsonhero, which allows you to view JSON content in a structured way, and even customizes views for some special format strings: ⁃ Preview URL content (multimedia can be played directly, JSON can be further explored) ⁃ Time fields (ISO8601) can display calendars ⁃ RGB Hex can preview colors ⁃ Embedded JSON strings can also be previewed in a structured way (this is the most practical!) They have also just launched a Chrome plugin. On the JSON data page (the content-type of the current page's URL is application/json), clicking on the plugin will take you to jsonhero to view.
I usually use the command line tool jless (similar to jq, but can implement folding) to preview JSON. curl https://example.com/demo.json | jless A common scenario is to copy curl from Chrome DevTool and call it for preview, so it's not convenient to use jsonhero directly at this time. So I wrote a command line tool similar to jq https://github.com/sorcererxw/jsonhero to implement outputting JSON to jsonhero for viewing. go install github.com/sorcererxw/jsonhero@latest curl https://example.com/demo.json | jsonhero
June 9, 2022
I discovered a gem on Product Hunt, jsonhero, which allows you to view JSON content in a structured way, and even customizes views for some special format strings: ⁃ Preview URL content (multimedia can be played directly, JSON can be further explored) ⁃ Time fields (ISO8601) can display calendars ⁃ RGB Hex can preview colors ⁃ Embedded JSON strings can also be previewed in a structured way (this is the most practical!) They have also just launched a Chrome plugin. On the JSON data page (the content-type of the current page's URL is application/json), clicking on the plugin will take you to jsonhero to view.
June 9, 2022
https://buf.build/blog/connect-a-better-grpc The Buf team, a Protobuf management tool, has released an RPC suite called Connect. They listed some problems with gRPC: ⁃ Too complex, hard to debug, and the large codebase is prone to vulnerabilities ⁃ Does not use the net/http standard library ⁃ Does not support browsers Connect is an optimization of gRPC, with some features: ⁃ Simplified code, including more readable generated code ⁃ Uses the net/http standard library, better compatibility ⁃ Supports three protocols: gRPC/gRPC-Web/Connect ⁃ Only supports HTTP POST method, supports both HTTP/1.1 and HTTP/2, and both pb and json data formats ⁃ Supports the full gRPC protocol, including server reflection and health checks. Compared to Twitch's twirp, Connect is still compatible with the gRPC protocol, while twirp is more like a JSON-RPC based on the Protobuf generator. It seems that Connect is indeed "A better gRPC", which can cater to high-performance scenarios and also do fallback for restricted environments (browsers/debugging).
June 4, 2022
A common situation at work: Using various strong typing tools such as Protobuf/Thrift/Golang, but for the sake of flexibility, all kinds of fields are wrapped in an object and serialized into strings for transmission. In order to parse this object, various utils are written to extract the values inside, which is truly puzzling behavior.
May 31, 2022
The stacked design of Project Volterra looks cool, can it infinitely expand computing power? Although I don't use Windows for development, it seems to be a good option for a home server. https://www.youtube.com/watch?v=yICVNta8jMU
May 27, 2022
The concept of Bionic Reading has gained popularity again recently, presumably because a team shared their Bionic Reading Chrome Extension named Jiffy Reader on HN. https://news.ycombinator.com/item?id=31475420 My first encounter with Bionic Reading was on Reeder. I personally think the principle is roughly as follows: bold and darken the first few letters of each word in proportion, allowing the eyes to quickly locate between words, improving efficiency and avoiding distraction. It indeed works well for reading English articles. However, Jiffy Reader seems to have not been released yet. I found another Bionic Reading plugin in the Chrome store, and the experience is still good. You can customize the weight and proportion of highlighted letters. Recommended. https://chrome.google.com/webstore/detail/bionic-reading-digest-pas/lbmambbnglofgbcaphmokiadbfdicddj/related
May 24, 2022
I can totally empathize with the feeling of having to write a document and hold a meeting even for the slightest thing in a large tech organization. https://liou28335.medium.com/many-software-companies-are-a-joke-9f4b10378c7a
May 24, 2022
Just saw this on HN https://indigostack.app/ One-click local development environment setup, including reverse proxy configuration SSL/database, etc., can indeed save a lot of work. The interface displays all components as a host on a rack, which is quite interesting. I've tested it and found that the software package is very large, about 1.6GB, I guess all dependent software is packaged in. Also, since it has not been officially released, there are still bugs, it cannot be used for actual production.
May 24, 2022
#golang From what I've seen, the Go backend of ByteDance is a thorough polyrepo, where one repository is only one service, and the common logic of multiple services is broken down into various packages for reuse. In my opinion, it's not as convenient and efficient as monorepo, but I can't deny that polyrepo can simplify the work of permission control, CI, and other infrastructure. There is no absolute correct choice, it depends on different organizational structures. If you choose polyrepo, in daily development, there often arises a situation where one requirement needs to change multiple repos. If there are go mod dependencies, you need to first push the dependencies to the repository, and then pull the new version in the caller repository through git commit hash. It can be considered quite troublesome. Fortunately, Go 1.18 finally added the workspace feature, which allows you to directly point a package to another directory through go.work locally. The code changes on one side can be called immediately on the other side, greatly reducing the trouble of cross-repository development. . └── gitlab/ ├── biz_1/ │ ├── svc_1/ │ │ └── go.mod │ └── svc_2/ │ └── go.mod ├── biz_2/ │ └── svc_3/ │ └── go.mod ├── common/ │ ├── pkg_1/ │ │ └── go.mod │ └── pkg_2/ │ └── go.mod └── go.work In this way, you only need to open the entire code root directory with IDE and configure go.work, and you can immediately get a development experience almost the same as monorepo!
May 4, 2022
Although the function of publicly displaying IP addresses in posts is not worth making a fuss about, the intention behind it is somewhat unsettling. It's unclear whether this will spread to all platforms in the future, but adding rules to the proxy can be considered a form of protest.
April 30, 2022
HTML embed is a very interesting feature that allows users to embed a certain dynamic content into any third-party website for display. For the convenience of visitors, the source website must allow the embed resources to support anonymous access. This is simply a paradise for web crawlers. You only need to infer the embed URL through the dynamic ID to crawl the dynamic content. Interestingly, in the overseas Internet ecosystem, embed is an important traffic entrance, and many websites will provide such a feature. Even anti-crawling platforms like Instagram can crawl dynamic content in this way. Of course, this is not insurmountable. For example, Linkedin has separated the embed ID and dynamic ID. Users can only get the embed URL through the dynamic ID within Linkedin, which can prevent web crawlers from massively crawling data.
April 24, 2022
https://danpetrov.xyz/programming/2021/12/30/telegram-google-translate.html This article is quite interesting, it talks about Telegram introducing a translation feature based on Google Translate, but using a private API to take advantage of Google. This API is used to implement text translation for the Chrome browser, provided free of charge to users, and naturally can be accessed anonymously. The caller only needs to simulate a request in various ways, deceiving Google into thinking that this is a request from an ordinary user.
April 23, 2022
Because Go's error does not carry an error stack, if you want to trace the source of the error, the native method is to wrap each error throw with a message. Because this method is too troublesome, I have always used pkg/errors to encapsulate the original error stack in the error. However, this inevitably requires me to customize the final error collection and log output to implement monitoring. Now it's fine, just let Copilot generate the wrap message. Copilot makes "simplicity is the ultimate sophistication" really simple and easy to use.
April 23, 2022
Demystifying gRPC-web, all gRPC-web requests are sent using POST, which means all read requests cannot be cached by the browser/CDN, which has a significant impact on performance. Refer to https://github.com/grpc/grpc/issues/7945 Currently, it seems that wrapping a layer on the client side, recognizing the request body and caching can alleviate this, but after all, it is not a native implementation of the HTTP protocol, and it is not elegant. However, generating Client and Server Stub based on IDL is really cool, and it seems that I will continue to use gRPC web for now.
April 21, 2022
Recently, Byte's Soda Music was released, and I wanted to give it a try. I plan to run a script first to sync my playlist on Spotify over. Unlike Moo which only supports WeChat login, Soda Music provides a way to log in with a mobile number, which gives the opportunity to obtain a token through a script. I tried to use Proxyman to capture the details of each interface, but found that as soon as MITM was started, Soda could not work properly. I guess the client has enabled SSL pinning, which prohibits man-in-the-middle attacks. After some tinkering, I finally bypassed SSL pinning. I tried to capture the login interface, and found that some key fields in the request parameters were encrypted... Although client encryption can definitely be cracked, it's really too much trouble, so I gave up.
April 21, 2022
I've tried out arctype, another fundamental tool based on the "collaboration + X" concept. As a local SQL Client, its quality is quite impressive. It responds faster than DataGrip and has a better interface than Sequel Ace. In addition, arctype also supports direct connection to PlanetScale databases (without the need for local port forwarding), which is very user-friendly for PlanetScale users (including myself).
April 21, 2022
Struggling with using ffmpeg on Vercel Serverless functions: - Unable to pre-install ffmpeg (configuring lambda layer seems troublesome) - Limit on the size of the compiled product, after compression (tar.gz), it needs to be controlled within 50MB In Node, you can use ffmpeg-static, but there is no good solution for Go. So recently I encapsulated a library github.com/go-ffstatic/ffstatic. A solution similar to ffmpeg-static———the library contains the complete ffmpeg executable. However, to make it easier to use, I directly embed the entire ffmpeg into the Go compiled product and export them to the tmp directory when starting. However, there are still some minor issues. The x64 versions of ffmpeg and ffprobe in ffmpeg 4.x are both 70+MB. When packaged together, the compressed size is still 50+MB, exceeding the limit of Vercel. It's unlikely to compile a streamlined ffmpeg by myself. Currently considering using ffmpeg 3.x version to reduce the size. In addition, you can also consider whether to package ffprobe through compilation parameters.
April 21, 2022
Recently, I switched the proxy for bypassing internet restrictions on my Merlin router from fancyss to MerlinClash. Not only has the stability improved significantly, but it also now supports the use of Clash rules. Highly recommended! https://t.me/merlinclashcat
April 21, 2022
I studied the Go generic tool library github.com/samber/lo, and saw this piece of code... It's fair to say I was a bit stunned. This is why so many people were against adding generics to Go before.
April 21, 2022