Many teams don't test their functions separately. They run the whole project to see the result. When something goes wrong, they check the log and use a debugger. They are the majority, at least from my experience. Static type checking and type annotations are efficient for these teams because type annotations give a rough idea about data for each function. They can't look at testing data, which doesn't exist.
Still, I wonder if forcing type annotation is practical. Reading a long function is difficult. By splitting a long function, I found that type annotation can be distracting because instead of focusing on logic, I have to think about type annotation; sometimes, the size of type annotation is about half the function's size. Maybe forcing type annotation only on functions, which an outsider from another module can call, like in OCaml, is practical. I haven't coded in OCaml beyond some toy programs. So I don't know if it is really practical as I imagine.
My computer has many cores but doesn't have enough RAM to store the whole data. So I usually need to process data from a stream parallelly, for example, reading a Thai text line by line and dispatch them to processors running on each core.
In Clojure, I can use pmap because pmap works on a lazy sequence. However, using Lparallel's pmap on Common Lisp with lazy sequences wasn't in the example. So I used Bordeaux threads and ChanL channels instead. It worked. Still, I had to write repetitive code to handle threads and channels whenever I wanted to process the stream parallelly. It didn't only look messy, but it came with many bugs.
So I created a small library, called stream-par-procs, to wrap threads and channels management. As shown in the diagram, the reader reads a line for the stream, the system dispatch a line to different processors, and finally, the collector creates the final result.
I only need to define a function to be run in the processor, another function for running in the collector, and other functions for initializing states. It hides details, for example, joining collector thread when every processor sent END-OF-STREAM.
In brief, stream-par-procs makes processing a stream parallelly in Common Lisp more convenient with hopefully fewer bugs by reusing threads and channels management code.
Fetching data from REST APIs or a plain HTTP server is usually a part of my work. Many problems may arise during fetching data; for example, a server goes offline, my storage is out of space, an API gives the wrong URL, a server sends data in an invalid format, etc. To illustrate the situation, I can oversimplify my task to the function below:
deffetch(uri):requests.get(uri)
A fetcher usually faces one of these problems. I don't want to rerun a fetcher after running it for an hour or a day. Maybe I can write my fetcher to resume working after it stops at any state, but it will take time, and my fetcher will be more complex. If I code in Python, the fetch function can raise many exceptions, and my fetcher will stop because my code didn't handle any exceptions.
I see two types of environments that will help me fix the problem.
Type checker may help me handle as many types of exceptions as possible upfront. Java definitely can do this because, in Java, I have to declare the list of exceptions as a part of the fetch function, and Javac checks if the caller handles these exceptions. Rust doesn't use a Java-like exception, but the result type encodes the types of errors, so the Rust compiler can check if a caller handles these errors.
Common Lisp runtime will run a debugger if unhandled errors occur. So I can choose what to do next. For example, I can empty my trash folder if my storage is out of space. I still don't know if I can handle the storage space problem later with Erlang runtime.
Python doesn't seem to belong to (1) nor (2). I tried mypy, a static type checker for Python, but it didn't warn me anything. And I don't know how to retry or do something else with Python's exception system rather than let it stop.
I ran the average-1000-etipitaka-flexi-streams on SBCL 2.2.5-1.1-suse on my laptop with Celeron N4500. It took 1.591 seconds.
Then I change the file to my-data.ndjson.zst, whose average line length is 515.5 bytes. Running average-1000-ndjson-flexi-streams took 4.411 seconds.
So I also tested with my customized utf8-input-stream. Running average-1000-etipitaka-utf8-input-stream, and average-1000-ndjson-utf8-input-stream took 0.019 seconds, and 0.043 seconds respectively, which means utf8-input-stream is 83X faster for short lines, and 102X faster for long lines, than Flexi-streams in these tests.
I have a big line-separated JSON file, compressed in Zstd format. I will call this file data.ndjson.zst.
I want my program to read the file line by line. I can use https://github.com/glv2/cl-zstd to make a binary stream from the file. Still, I can run the function read-line of a binary stream. So I need to wrap the binary stream with Flexi-stream.
Printing those lines is a realistic example. So you can replace (print line) with your practical applications, for example, parse a JSON line and extract a book title.
I want to use Emacs 28.1 on aging OS that I mustn't modify /usr. I tried Docker but I want to use local command too. SBCL didn't work properly on old Docker. So I install Emacs from a source tarball to my home directory.
However, Emacs failed to verify TLS certs. So I installed GNUTLS, Nettle, Idn, and Unistring.
#!/bin/bashexport PKG_CONFIG_PATH=$HOME/lib64/pkgconfig:$HOME/lib/pkgconfig
export LD_RUN_PATH=$HOME/lib:$HOME/lib64
export LD_LIBRARY_PATH=$HOME/lib:$HOME/lib64
export LDFLAGS="-L$HOME/lib -L$HOME/lib64"rm-rf libunistring-1.0 libidn2-2.3.2 nettle-3.6 gmp-6.2.1 emacs-28.1
curl https://ftp.gnu.org/gnu/libunistring/libunistring-1.0.tar.gz | tar-xzvf - &&\pushd libunistring-1.0 &&\
./configure --prefix=$HOME&&\
make -j`nproc`&&\
make install&&\popd||exit 1
curl https://ftp.gnu.org/gnu/libidn/libidn2-2.3.2.tar.gz | tar-xzvf - &&\pushd libidn2-2.3.2 &&\
./configure --prefix=$HOME&&\
make -j`nproc`&&\
make install&&\popd||exit 1
curl https://ftp.gnu.org/gnu/nettle/nettle-3.6.tar.gz | tar xzvf - &&\pushd nettle-3.6 &&\
./configure --prefix=$HOME--enable-mini-gmp&&\
make -j`nproc`&&\
make install&&\popd||exit 1
curl https://www.gnupg.org/ftp/gcrypt/gnutls/v3.7/gnutls-3.7.6.tar.xz | tar xJvf - &&\pushd gnutls-3.7.6 &&\
./configure --prefix=$HOME&&\
make -j`nproc`&&\
make install&&\cd .. ||exit 1
curl http://ftp.gnu.org/pub/gnu/emacs/emacs-28.1.tar.gz | tar-xzvf - &&pushd emacs-28.1 &&\
./configure --prefix=$HOME--with-x-toolkit=no --with-xpm=no --with-jpeg=no --with-jpeg=no --with-gif=no --with-tiff=no --with-png=no &&\
make -j`nproc`&&\
make install&&\cd .. ||exit 1
echo
echo
echo"Emacs 28.1 must be ready!"echo
The worst thing I did in Rust was using unbounded asynchronous channels. It took days to figure out that my program had backpressure, and channels ate up all RAM.
So, since then, I have always used sync_channel or bounded channels from Crossbeam.
I was in a world where a code describing type signature was longer than a code telling what a program does. I don't want to bring that situation back.
I expect AI code completion to replace static typing. Ruby Type Signature (RBS) and similar things must be inferred automatically from a database or other data sources.
; caught WARNING:
; Derived type of STATIC1::A is
; (VALUES FIXNUM &OPTIONAL),
; conflicting with its asserted type
; STRING.
; See also:
; The SBCL Manual, Node "Handling of Types"
So SBCL - a Common Lisp implementation, can check type in compile-time. Anyway, a programmer needs to read warnings.
Last week, I watched a video about coding and Docker configuration on a TV. I couldn't read any line of code. Then I thought about visually impaired people. How do they code? Every student in Thailand must learn to code. I presume the situation is similar in every country.
People widely use text-to-speech services these days. I cannot find any text-to-speech service for reading source code aloud. Let's assume we have a modified version of a text-to-speech service.
So I looked at source codes in different programming languages on the CodeRosetta website. I perceive Python code blocks by their visual structure purely. To read Python source code, I have to encode its visual structure, namely indents to sounds. Reading a nested code block won't be easy to understand. For example, reading twelve leading white spaces aloud will be very strange. In Lisp, reading open parenthesis and close parenthesis is more straightforward, but I will forget which parenthesis. So the best form of code blocks is in QuickBasic, which has different keywords between different kinds of blocks. For example, FOR with NEXT, and IF with END IF. Later I got a comment from Lemmy.ml, which told me that Ada also has different keywords between different kinds of blocks. Another idea from Lemmy.ml is the reader must convert the Python code block into a similar form as Ada or QuickBasic before reading.
MBasic refers to code by line numbers instead of code blocks. However, by listening to five lines, I forgot the line number. For example, when I heard gosub 70, I forgot what was at line 70.
In X86 Assembly, a programmer labels only the line that the program will jump to it. So X86 Assembly code looks much better than MBasic.
Still, coding in X86 Assembly can be exhaustive in many cases. For example, X86 Assembly doesn’t support recursion. Writing quick sort in X86 Assembly can be too difficult for learning to code.
Haskell doesn’t rely on code blocks. However, reading the symbols, for example, >>= is challenging. Prolog’s symbols are easy to read. For example, we can read :- as IF. Anyway, the Prolog programming paradigm is different from the mainstream one now. So Erlang, whose syntax is similar to Prolog, is a more practical alternative.
In brief, Erlang is a practical, less visual-centric program language. Because it mostly relies on names instead of code blocks, reading names aloud is much easier than reading code blocks aloud. Furthermore, the Erlang programming paradigm is more mainstream now.
#################### F1 ###################
Evaluation count : 6 in 6 samples of 1 calls.
Execution time sample mean : 3.811858 sec
Execution time mean : 3.812064 sec
#################### F2 ###################
Evaluation count : 6 in 6 samples of 1 calls.
Execution time sample mean : 1.490624 sec
Execution time mean : 1.490777 sec
F1, which is the lazy sequence version, took 3.812064 seconds. F2, which is the transducer version, took 1.490777. So the transducer version is 155.71% faster than the lazy sequence version.
In brief, this biased experiment shows the transducer version is much faster than the pure lazy sequence version.
I recommend Clojure as the first programming language to learn for beginners. I recommend "Poetry of Programming" as the first book for learning Clojure. I recommend "Clojure for the Brave and True" as the second book for learning Clojure.
Lisp taught me to create functions that transform data step by step instead of adding mutable variables into loops. An immutable data structure is important for transforming data step by step without messing up existing data. Lisp's default data structure is a singly linked list.
Lisp macro is easy to write compared to competitors. Rust's macro is powerful. Rust's macro is about transforming a TokenStream, and #Lisp's macro is about transforming a list. However, I prefer transforming a list to a TokenStream since I don't have to learn new structures and new commands for manipulating a new structure.
And it has too many "return" statements. Anyway, I suppose Python doesn’t prefer a recursion. So this isn’t the case when we code in the Python way.
The code above concatenate Python's lists instead of creating a linked list like in Lisp.
So I made this experiment.
importtimen=100# Concatinating vectors
v=[]t1=time.time_ns()foriinrange(n):v=[i]+vt2=time.time_ns()# Creating cons cells for a singly link list
l=Nonet3=time.time_ns()foriinrange(n):l=(i,l)# Creating a cons cell
t4=time.time_ns()print((t2-t1)/(t4-t3))
At n = 100, concatenating vectors require 2 times of running time compared to creating cons cells for a linked list. So I keep using linked lists.
I want to install a specific version of RocksDB across different machines with different operating systems. Thus I can't rely on APT or other package management systems. I followed INSTALL.md by running make on the bundled Makefile, but last Saturday, I found that RocksDB couldn't read Snappy compressed data, although I installed the "libsnappy-dev" package. I tried many different ways to enable Snappy support. Then I decided to use CMake, which appeared only once in INSTALL.md. Now it works. My install script looks like this:
แต่ถ้า Fields มีเยอะ ๆ สัก 10 Fields ขึ้นไป Code ก็จะเริ่มดูยากยาวมาก การใช้ Pydantic model เข้ามาช่วยก็จะจัดการง่ายขึ้น ปกติเราใช้ Pydantic model ใน Body แบบ JSON กันอยู่แล้วแต่สำหรับ Form Data จะมีลูกเล่นนิดหนึ่ง ค้นหาใน internet ไปเจอวิธีที่คิดว่าง่ายที่สุดละเลยบันทึกไว้สักหน่อย
// define variablesletcamera_id='';// current camera idletstream=null;// current streamletdevices=[];// all cameraconstdiv=document.createElement('div');// parent div playgroundconstcamera_list=document.createElement('select');// camera dropdownconstcamera_label=document.createElement('h3');// current selected cameraconstvideo=document.createElement('video');// video object// scan all camerasasyncfunctionscan_cameras(){letcamera_found=false;awaitnavigator.mediaDevices.enumerateDevices().then((deviceInfos)=>{deviceInfos.forEach((deviceInfo)=>{if(deviceInfo.kind==='videoinput'){if(!camera_found){camera_label.textContent=`Camera: ${deviceInfo.label}`;camera_id=deviceInfo.deviceIdcamera_found=true}devices.push(deviceInfo)}})})div.appendChild(camera_label);}// change camera by idasyncfunctionchange_camera(e){camera_id=e.target.value;camera_label.textContent=`Camera: ${devices.filter(i=>i.deviceId==camera_id)[0].label}`;awaitstart_camera();}// stop all active cameraasyncfunctionstop_camera(){if(stream){awaitstream.getVideoTracks().forEach(track=>track.stop());}}// start camera by selected idasyncfunctionstart_camera(){awaitstop_camera();stream=awaitnavigator.mediaDevices.getUserMedia({video:{deviceId:{exact:camera_id},audio:false}});video.srcObject=stream;awaitvideo.play();}// create camera list dropdownfunctioncreate_camera_select(){camera_list.appendChild(newOption("Select Camera",0))devices.forEach((device,i)=>{camera_list.appendChild(newOption(device.label||`Camera ${i+1}`,device.deviceId));})div.appendChild(camera_list);camera_list.addEventListener("change",change_camera);}// take photoasyncfunctiontakePhoto(quality){awaitscan_cameras();create_camera_select();constcapture=document.createElement('button');capture.textContent='Capture';div.appendChild(capture);video.style.display='block';document.body.appendChild(div);div.appendChild(video);awaitstart_camera();google.colab.output.setIframeHeight(document.documentElement.scrollHeight,true);// Wait for Capture to be clicked.awaitnewPromise((resolve)=>capture.onclick=resolve);constcanvas=document.createElement('canvas');canvas.width=video.videoWidth;canvas.height=video.videoHeight;canvas.getContext('2d').drawImage(video,0,0);awaitstop_camera();div.remove();returncanvas.toDataURL('image/jpeg',quality);}
โปรแกรมตัดเกรดเป็นตัวอย่างของ business logic ที่ดีเลยครับ เพราะว่าหลายครั้งก็มีการเปลี่ยนแปลงกฎเกณฑ์ วิธีคิดเกรดของแต่ละสถาบันไม่เหมือนกันก็ได้ บางวิชาในสถาบันเดียวกันก็ตัดเกรดไม่เหมือนกัน บางมีชามี A B C D F บางวิชามีเกรดแค่ S กับ U
I agree that using JavaScript increases the chance of participation. I released a few versions of Thai word breakers in different programming languages. One on node.js is the most popular. 8 people contributed to the JS-based project compared to 2-3 people in other programming languages. However, JS has a downside too. In 2017, @iporsut and I made an experiment to compare Thai word breakers that we created. JS version running time is 15X of the Rust version. Even by comparing with another dynamic language, the Julia version is faster than the one in JS.
I created a website using node.js in 2014, and it is still running. The performance is good. However, I have a few regrets.
We had a very hard time by install this project on other team members who use Windows 10 because we didn't know how to build a Bcrypt library.
Recently, I have to fix the project without adding any new feature because Express.js was changed, MongoDB was changed, and some packages that I used were abandoned.
It was a small project so I wanted to keep the session storage in RAM, but I can't since I ran 4 node.js processes. Now the project requires Redis as session storage, which causes more troubles for team members, who don't familiar with GNU/Linux, Docker, or WSL.
I knew I’m in a dream. It was a bad dream. I wanted to get out I knew I’m in a dream. I decided to commit a suicide so it would wake me up. I jumped out of a window, closed my eyes. It was pitch black. Eyes opened, I woke up .. .. in … Continue reading inception ?→
A few days ago, I couldn't get a WebSocket library working with another library on Python 3.10. So to avoid those dependencies, I implemented my WebSocket client on a low-level socket API. I implemented one in Common Lisp first. Then I translated it to Python. My WebSocket client is very far away from being completed, but at least it can run.
The first part is import. I imported only three things that are socket, ssl, and uuid.
importsocketimportsslimportuuid
The second part is based on HTTP headers for telling the server to switch to the WebSocket mode.
The third part is reading a content length. I assume that the server keep sending text contents. In read_payload_len(), s.recv(1) read a byte from the socket s. & 0x7F is for masking only 7 bits. If length is 127, the length is in the next 4 bytes instead. If length is 126, the length is in the next 2 bytes. Otherwise the function just returns the length.
In the fourth part, we read a web socket frame. The program determine a frame type from opcode, which in the last 4 bits of the header. I should implement PING-PONG part but I didn't. According to RFC6455, which I forgot to mention before, opcode == 0x1 means the frame is a text frame. So the program reads payload length and reads the payload.
The last part is for opening connections and SSL/TLS wrapper. The function created a socket and wrapped it with TLS/SSL wrapper. The sending upgrade message to ask the server to switch to Websocket mode and the keep reading frames.
