Pthreads based program crashing on MacOS doesn't crash on Linux - pthreads

I have a pthreads based program with 3 threads that crashes on MacOS (Darwin Kernel Version 19.6.0) within a few seconds. I was expecting the same behavior on Linux. But the program doesn't crash on Linux. I am left speculating if Linux has a different thread scheduling policy or if it is something else. Any pointers appreciated.
This is the program. If I don't use pthread mutex lock in the printer thread, it is supposed to crash because the linked list is left in an inconsistent state.
#include<stdio.h>
#include<string.h>
#include<pthread.h>
#include<stdlib.h>
#include<unistd.h>
#include "list.h"
#define RANGE_MIN 1
#define RANGE_MAX 10
pthread_t tid[3];
list_t list;
pthread_mutex_t queue_lock;
/* Return a uniformly random number in the range [low,high]. */
int random_range (unsigned const low, unsigned const high)
{
unsigned const range = high - low + 1;
return low + (int) (((double) range) * rand() / (RAND_MAX + 1.0));
}
/*
* consumer thread function
*/
void* consumer(void *arg)
{
static int val = 0;
while (1) {
//sleep(1);
pthread_mutex_lock(&queue_lock);
while (list.total_elem > 0)
consume_and_delete(&list);
pthread_mutex_unlock(&queue_lock);
}
}
void *producer(void *arg)
{
while (1) {
//sleep(1);
pthread_mutex_lock(&queue_lock);
while (list.total_elem < 10)
add_to_list(&list, random_range(RANGE_MIN, RANGE_MAX));
pthread_mutex_unlock(&queue_lock);
}
}
/*
* printer thread function
*/
void* printer(void *arg)
{
while (1) {
//pthread_mutex_lock(&queue_lock); //lines deliberately commented out to show the crash
print_list(&list);
//pthread_mutex_unlock(&queue_lock);
}
}
int main(void)
{
int ret;
if (pthread_mutex_init(&queue_lock, NULL) != 0) {
printf("\n mutex init failed\n");
return 1;
}
ret = pthread_create(&(tid[1]), NULL, &consumer, NULL);
if (ret != 0) {
printf("\ncan't create thread :[%s]", strerror(ret));
}
ret = pthread_create(&(tid[0]), NULL, &producer, NULL);
if (ret != 0) {
printf("\ncan't create thread :[%s]", strerror(ret));
}
ret = pthread_create(&(tid[2]), NULL, &printer, NULL);
if (ret != 0) {
printf("\ncan't create thread :[%s]", strerror(ret));
}
pthread_join(tid[0], NULL);
pthread_join(tid[1], NULL);
pthread_join(tid[2], NULL);
pthread_mutex_destroy(&queue_lock);
return 0;
}
For the sake of completeness, here is the code for list.h and list.c
ifndef __LIST_H__
#define __LIST_H__
typedef struct node_ {
int num;
struct node_ *next;
} node_t;
typedef struct list_ {
int total_elem;
node_t *head;
} list_t;
#define TRUE 1
#define FALSE 0
void add_to_list(list_t *list, int num);
node_t *allocate_new(int num);
void consume_and_delete(list_t *list);
void print_list(list_t *list);
#endif
list.c:
#include <stdio.h>
#include <stdlib.h>
#include "list.h"
void add_to_list(list_t *list, int val)
{
node_t *tmp;
node_t *new;
new = allocate_new(val);
if (!new) {
printf("%s: allocation failure \n", __FUNCTION__);
}
printf("%s: Enqueueing %d\n", __FUNCTION__, new->num);
if (!list) {
return;
}
if (!list->head) {
list->total_elem++;
list->head = new;
return;
}
tmp = list->head;
while (tmp->next) {
tmp = tmp->next;
}
tmp->next = new;
list->total_elem++;
}
node_t *allocate_new(int num)
{
node_t *tmp;
tmp = malloc(sizeof(node_t));
if (!tmp) {
printf("%s: failed in malloc\n", __FUNCTION__);
return NULL;
}
tmp->num = num;
tmp->next = NULL;
return tmp;
}
/* reads from the front of the queue and deletes the element
*/
void consume_and_delete(list_t *list)
{
node_t *tmp, *next;
if (!list->head) {
return;
}
tmp = list->head;
next = tmp->next;
printf("%s: Dequeueing %d\n", __FUNCTION__,tmp->num);
list->head = next;
list->total_elem--;
free(tmp);
}
void print_list(list_t *list)
{
node_t *tmp;
if (!list->head) {
printf("%s: queue empty \n", __FUNCTION__);
}
tmp = list->head;
while (tmp) {
printf("%d ", tmp->num);
tmp = tmp->next;
}
printf("\n");
}
Makefile:
all: prodcon
prodcon: list.o prod_consume.o
gcc -g -o prodcon list.o prod_consume.o -lpthread
list.o: list.c list.h
gcc -g -c -o list.o list.c
prod_consume.o: prod_consume.c list.h
gcc -g -c -o prod_consume.o prod_consume.c
clean:
rm -f *.o ./prodcon
Note that if I un-comment pthread_mutex_lock and unlock calls in printer fn, the program runs without a crash on MacOS, as expected. But on Linux, even without un-commenting those lines in printer thread, it runs fine.
So my question. Is thread scheduling different in Linux. Or is is there some other reason?
Any reason the program runs fine on Linux, while it crahes on MacOS?

//lines deliberately commented out to show the crash
The print_list accesses the list without the lock; of course the code will intermittently crash, what did you expect?
on Linux, even without un-commenting those lines in printer thread, it runs fine.
It doesn't "run fine" -- it exercises undefined behavior, and will crash if you run enough times and the stars align to expose the data race.

Related

pthread_t declarations need to be global?

[Edited and added reprex] When I put the declarations below in the beginning of the main() function, the behavior of the threads is erratic and incorrect. When I make these declarations global, or not the first declarations in main(), everything works fine. I'm using mingw-w64 on Windows 10.
pthread_t thr1, thr2, thr3;
Below is the program, it spawns three threads to read integers from three different text files and add them all to a single global variable. The call to pthread_join() for thr3 always fails with error code 3, and the final result of sum is different in different runs. But it all works fine if the pthread_t declarations are in a different location. I hope this is short enough it's less than 70 lines including whitespace.
#include <stdlib.h>
#include <pthread.h>
#include <stdio.h>
long sum = 0;
pthread_mutex_t sumLock = PTHREAD_MUTEX_INITIALIZER;
void* thrFunc(void* arg)
{
char curLine[50];
FILE *inFile = fopen((char*)arg, "r");
long curNum;
if (!inFile) {
printf("Error opening input file %s\n", arg);
pthread_exit((void*)1);
}
while (fgets(curLine, sizeof(curLine), inFile)) {
curNum = strtol(curLine, 0, 0);
pthread_mutex_lock(&sumLock);
sum += curNum;
pthread_mutex_unlock(&sumLock);
}
fclose(inFile);
pthread_exit((void*)0);
}
int main(void)
{
pthread_t thr1, thr2, thr3;
long threadResult;
int err;
if (pthread_create(&thr1, 0, thrFunc, (void*)"C:\\Users\\paulc\\long1.txt")) {
printf("Failed to create thread thr1\n");
exit(1);
}
if (pthread_create(&thr2, 0, thrFunc, (void*)"C:\\Users\\paulc\\long2.txt")) {
printf("Failed to create thread thr2\n");
exit(1);
}
if (pthread_create(&thr3, 0, thrFunc, (void*)"C:\\Users\\paulc\\long3.txt")) {
printf("Failed to create thread thr3\n");
exit(1);
}
if (err = pthread_join(thr1, (void**)&threadResult)) {
printf("failed to join thr1, error code = %d\n", err);
}
if (err = pthread_join(thr2, (void**)&threadResult)) {
printf("failed to join thr2, error code = %d\n", err);
}
if (err = pthread_join(thr3, (void**)&threadResult)) {
printf("failed to join thr3, error code = %d\n", err);
}
printf("Total: %ld\n", sum);
}

Create C timer in macOS

I would like to know how to create a timer in macOS / iOS. In linux you can create it using timer_create() function of time.h class but in macOS / iOS this function don't exist.
Something like NSTimer (objective-c) but in C.
Thanks
After updating the link for Grand Central Dispatch timers (Apple's page) on a previous answer, I created some example code for two timers. It should be noted that this works on FreeBSD and MacOS, but not Linux (without GCD support). The example creates two event timers, one that fires 0.2sec and one that fires 0.5sec over a total of 20 times. There is a 1 second delay that exists before execution starts. The sleep() functions are not used.
#include <dispatch/dispatch.h>
#include <stdio.h>
#include <stdlib.h>
int i = 0;
dispatch_queue_t queue;
dispatch_source_t timer1;
dispatch_source_t timer2;
void sigtrap(int sig)
{
dispatch_source_cancel(timer1);
dispatch_source_cancel(timer2);
printf("CTRL-C received, exiting program\n");
exit(EXIT_SUCCESS);
}
void vector1(dispatch_source_t timer)
{
printf("a: %d\n", i);
i++;
if (i >= 20)
{
dispatch_source_cancel(timer);
}
}
void vector2(dispatch_source_t timer)
{
printf("b: %d\n", i);
i++;
if (i >= 20) //at 20 count cancel the
{
dispatch_source_cancel(timer);
}
}
int main(int argc, const char* argv[]) {
signal(SIGINT, &sigtrap); //catch the cntl-c
queue = dispatch_queue_create("timerQueue", 0);
// Create dispatch timer source
timer1 = dispatch_source_create(DISPATCH_SOURCE_TYPE_TIMER, 0, 0, queue);
timer2 = dispatch_source_create(DISPATCH_SOURCE_TYPE_TIMER, 0, 0, queue);
// Set block for dispatch source when catched events
dispatch_source_set_event_handler(timer1, ^{vector1(timer1);});
dispatch_source_set_event_handler(timer2, ^{vector2(timer2);});
// Set block for dispatch source when cancel source
dispatch_source_set_cancel_handler(timer1, ^{
dispatch_release(timer1);
dispatch_release(queue);
printf("end\n");
exit(0);
});
dispatch_source_set_cancel_handler(timer2, ^{
dispatch_release(timer2);
dispatch_release(queue);
printf("end\n");
exit(0);
});
dispatch_time_t start = dispatch_time(DISPATCH_TIME_NOW, NSEC_PER_SEC); // after 1 sec
// Set timer
dispatch_source_set_timer(timer1, start, NSEC_PER_SEC / 5, 0); // 0.2 sec
dispatch_source_set_timer(timer2, start, NSEC_PER_SEC / 2, 0); // 0.5 sec
printf("start\n");
dispatch_resume(timer1);
dispatch_resume(timer2);
while(1)
{
;;
}
return 0;
}
You can use pthreads for macOS, with some combination of sleep and time
typedef struct Timer {
void (*fn)(void);
bool (*timer_delegate)(pthread_t, unsigned int, unsigned int);
unsigned int seconds;
} Timer;
void* timer_run(void *t) {
unsigned int start_time = time(NULL);
while(1) {
Timer tmr = *((Timer *) t);
bool should_kill_thread = tmr.timer_delegate(pthread_self(), start_time, time(NULL));
if (should_kill_thread) pthread_cancel(pthread_self());
tmr.fn();
sleep(tmr.seconds);
}
}
bool should_kill_thread(pthread_t t, unsigned int start_time, unsigned int utime_new) {
printf("the start time was %d and the new time is %d \n", start_time, utime_new);
if (utime_new - start_time >= 9) {
return true;
}
return false;
}
void hello_world() {
printf("%s\n", "Hello, World!");
}
int main(int argc, char const *argv[])
{
pthread_t t1;
Timer args;
args.fn = &hello_world;
args.timer_delegate = should_kill_thread;
args.seconds = 1; // call every 1 second
int id = pthread_create(&t1, NULL, timer_run, &args);
if (id) {
printf("ERROR; return code from pthread_create() is %d\n", id);
exit(EXIT_FAILURE);
}
pthread_join(t1, NULL); // blocks main thread
printf("%s\n", "DONE"); // never reached until t1 is killed
return 0;
}
Expanding on b_degnan's answer above (click that one), we had to (partially) support Linux code using timers, so we wrote a time.h that we could use directly.
#include <sys/stdtypes.h>
#include <stdbool.h>
#include <mach/boolean.h>
#include <sys/errno.h>
#include <stdlib.h>
#include <dispatch/dispatch.h>
#if !defined(MAC_OS_X_VERSION_10_12) || \
(MAC_OS_X_VERSION_MIN_REQUIRED < MAC_OS_X_VERSION_10_12)
typedef int clockid_t;
#endif
struct itimerspec {
struct timespec it_interval; /* timer period */
struct timespec it_value; /* timer expiration */
};
struct sigevent;
/* If used a lot, queue should probably be outside of this struct */
struct macos_timer {
dispatch_queue_t tim_queue;
dispatch_source_t tim_timer;
void (*tim_func)(union sigval);
void *tim_arg;
};
typedef struct macos_timer *timer_t;
static inline void
_timer_cancel(void *arg)
{
struct macos_timer *tim = (struct macos_timer *)arg;
dispatch_release(tim->tim_timer);
dispatch_release(tim->tim_queue);
tim->tim_timer = NULL;
tim->tim_queue = NULL;
free(tim);
}
static inline void
_timer_handler(void *arg)
{
struct macos_timer *tim = (struct macos_timer *)arg;
union sigval sv;
sv.sival_ptr = tim->tim_arg;
if (tim->tim_func != NULL)
tim->tim_func(sv);
}
static inline int
timer_create(clockid_t clockid, struct sigevent *sevp,
timer_t *timerid)
{
struct macos_timer *tim;
*timerid = NULL;
switch (clockid) {
case CLOCK_REALTIME:
/* What is implemented so far */
if (sevp->sigev_notify != SIGEV_THREAD) {
errno = ENOTSUP;
return (-1);
}
tim = (struct macos_timer *)
malloc(sizeof (struct macos_timer));
if (tim == NULL) {
errno = ENOMEM;
return (-1);
}
tim->tim_queue =
dispatch_queue_create("timerqueue",
0);
tim->tim_timer =
dispatch_source_create(DISPATCH_SOURCE_TYPE_TIMER,
0, 0, tim->tim_queue);
tim->tim_func = sevp->sigev_notify_function;
tim->tim_arg = sevp->sigev_value.sival_ptr;
*timerid = tim;
/* Opting to use pure C instead of Block versions */
dispatch_set_context(tim->tim_timer, tim);
dispatch_source_set_event_handler_f(tim->tim_timer,
_timer_handler);
dispatch_source_set_cancel_handler_f(tim->tim_timer,
_timer_cancel);
return (0);
default:
break;
}
errno = EINVAL;
return (-1);
}
static inline int
timer_settime(timer_t tim, int flags,
const struct itimerspec *its, struct itimerspec *remainvalue)
{
if (tim != NULL) {
/* Both zero, is disarm */
if (its->it_value.tv_sec == 0 &&
its->it_value.tv_nsec == 0) {
/* There's a comment about suspend count in Apple docs */
dispatch_suspend(tim->tim_timer);
return (0);
}
dispatch_time_t start;
start = dispatch_time(DISPATCH_TIME_NOW,
NSEC_PER_SEC * its->it_value.tv_sec +
its->it_value.tv_nsec);
dispatch_source_set_timer(tim->tim_timer, start,
NSEC_PER_SEC * its->it_value.tv_sec +
its->it_value.tv_nsec,
0);
dispatch_resume(tim->tim_timer);
}
return (0);
}
static inline int
timer_delete(timer_t tim)
{
/* Calls _timer_cancel() */
if (tim != NULL)
dispatch_source_cancel(tim->tim_timer);
return (0);
}
Or, as a gist:
https://gist.github.com/lundman/731d0d7d09eca072cd1224adb00d9b9e
I'd be happy to receive updates, if you enhance it further with more types. It can clearly be improved, but does just enough for our needs.
Went with straight C here, since it is a port of upstream source.

read event never triggered, using libevent

I just write an echo server using libevent, but it seems that the read event is never triggered. The code is:
#include <unistd.h>
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <netinet/tcp.h>
#include <event.h>
#include <event2/event.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <errno.h>
#include <fcntl.h>
#include <netinet/in.h>
#include <arpa/inet.h>
static short ListenPort = 19999;
static long ListenAddr = INADDR_ANY;//任意地址,值就是0
static int MaxConnections = 1024;
static int ServerSocket;
static struct event ServerEvent;
int SetNonblock(int fd)
{
int flags;
if ((flags = fcntl(fd, F_GETFL)) == -1) {
return -1;
}
if (fcntl(fd, F_SETFL, flags | O_NONBLOCK) == -1) {
return -1;
}
return 0;
}
void ServerRead(int fd, short ev, void *arg)
{
//1)when telnet on 1999 and send a string,this never prints,help!
printf("client readble\n");
fflush(stdout);
struct client *client = (struct client *)arg;
u_char buf[8196];
int len, wlen;
len = read(fd, buf, sizeof(buf));
if (len == 0) {
printf("disconnected\n");
close(fd);
event_del(&ServerEvent);
free(client);
return;
} else if (len < 0) {
printf("socket fail %s\n", strerror(errno));
close(fd);
event_del(&ServerEvent);
free(client);
return;
}
wlen = write(fd, buf, len);//1)client str never echo back
if (wlen < len) {
printf("not all data write back to client\n");
}
}
void ServerWrite(int fd, short ev, void *arg)
{
//2)to be simple,let writer do nothing
/* if(!arg)
{
printf("ServerWrite err!arg null\n");
return;
}
int len=strlen(arg);
if(len <= 0)
{
printf("ServerWrite err!len:%d\n",len);
return;
}
int wlen = write(fd, arg, len);
if (wlen<len) {
printf("not all data write back to client!wlen:%d len:%d \n",wlen,len);
}
*/
return;
}
void ServerAccept(int fd, short ev, void *arg)
{
int cfd;
struct sockaddr_in addr;
socklen_t addrlen = sizeof(addr);
int yes = 1;
cfd = accept(fd, (struct sockaddr *)&addr, &addrlen);
if (cfd == -1) {
//3)this prints ok
printf("accept(): can not accept client connection");
return;
}
if (SetNonblock(cfd) == -1) {
close(cfd);
return;
}
if (setsockopt(cfd, IPPROTO_TCP, TCP_NODELAY, &yes, sizeof(yes)) == -1) {
printf("setsockopt(): TCP_NODELAY %s\n", strerror(errno));
close(cfd);
return;
}
event_set(&ServerEvent, cfd, EV_READ | EV_PERSIST, ServerRead, NULL);
event_set(&ServerEvent, cfd, EV_WRITE| EV_PERSIST, ServerWrite,NULL);
event_add(&ServerEvent, NULL);
printf("Accepted connection from %s \n", (char *)inet_ntoa(addr.sin_addr));
}
int NewSocket(void)
{
struct sockaddr_in sa;
ServerSocket = socket(AF_INET, SOCK_STREAM, 0);
if (ServerSocket == -1) {
printf("socket(): can not create server socket\n");
return -1;
}
if (SetNonblock(ServerSocket) == -1) {
return -1;
}
memset(&sa, 0, sizeof(sa));
sa.sin_family = AF_INET;
sa.sin_port = htons(ListenPort);
sa.sin_addr.s_addr = htonl(ListenAddr);
if (bind(ServerSocket, (struct sockaddr*)&sa, sizeof(sa)) == -1) {
close(ServerSocket);
printf("bind(): can not bind server socket");
return -1;
}
if (listen(ServerSocket, MaxConnections) == -1) {
printf("listen(): can not listen server socket");
close(ServerSocket);
return -1;
}
event_set(&ServerEvent, ServerSocket, EV_READ | EV_PERSIST, ServerAccept, NULL);
if (event_add(&ServerEvent, 0) == -1) {
printf("event_add(): can not add accept event into libevent");
close(ServerSocket);
return -1;
}
return 0;
}
int main(int argc, char *argv[])
{
int retval;
event_init();
retval = NewSocket();
if (retval == -1) {
exit(-1);
}
event_dispatch();
return 0;
}
The server is tested using Telnet, but the client receives nothing.
The question details are posted as comments in the code above, at 1)、2)、3).
Can someone help me find out why the read event is never triggered?
basically you should not set the accepted socket as EV_WRITE until you actually want to write to the socket. You are telling libevent "let me know when I can write to this socket", which is pretty much always. So ServerWrite is being called in a tight loop. In practice the only time you need EV_WRITE is if you are writing a buffer but all of the bytes are not written. You then need to save the unwritten portion of the buffer and use EV_WRITE to signal when the socket can be written to again.

A core-dump when using lua_yield and lua_resume

I just want to resume the func coroutine twice, yield if n==0, and return if n==1 , but it core dumps, what't wrong with it?
the "hello world" should always be left in LL's stack, I can't figure out what is wrong.
[liangdong#cq01-clientbe-code00.vm.baidu.com lua]$ ./main
func_top=1 top=hello world
first_top=1 top_string=hello world
Segmentation fault (core dumped)
[liangdong#cq01-clientbe-code00.vm.baidu.com lua]$ cat main.c
#include "lua.h"
#include "lualib.h"
#include "lauxlib.h"
int n = 0;
int func(lua_State *L) {
printf("func_top=%d top=%s\n", lua_gettop(L), lua_tostring(L, -1));
if (!n) {
++ n;
return lua_yield(L, 1);
} else {
return 1;
}
}
int main(int argc, char* const argv[]) {
lua_State *L = luaL_newstate();
/* init lua library */
lua_pushcfunction(L, luaopen_base);
if (lua_pcall(L, 0, 0, 0) != 0) {
return 1;
}
lua_pushcfunction(L, luaopen_package);
if (lua_pcall(L, 0, 0, 0 ) != 0) {
return 2;
}
/* create the coroutine */
lua_State *LL = lua_newthread(L);
lua_pushcfunction(LL, func);
lua_pushstring(LL, "hello world");
/* first time resume */
if (lua_resume(LL, 1) == LUA_YIELD) {
printf("first_top=%d top_string=%s\n", lua_gettop(LL), lua_tostring(LL, -1));
/* twice resume */
if (lua_resume(LL, 1) == 0) {
printf("second_top=%d top_string=%s\n", lua_gettop(LL), lua_tostring(LL, -1));
}
}
lua_close(L);
return 0;
}
it core dumps in lua5.1, but works well in lua5.2 if change lua_resume(LL, 1) to lua_resume(LL, NULL, 1).
EDIT: I was actually totally wrong.
You cannot resume a C function.

A question of libevent example code: how is invoked?

I'm learning libev however the code is so hard to understand, so I choose to learn libevent first whose code is relatively clearer. But I encounter a problem when try the example (http://www.wangafu.net/~nickm/libevent-book/01_intro.html).
How is the code event_add(state->write_event, NULL) in do_read() make do_write() function invoked?
/* For sockaddr_in */
#include <netinet/in.h>
/* For socket functions */
#include <sys/socket.h>
/* For fcntl */
#include <fcntl.h>
#include <event2/event.h>
#include <assert.h>
#include <unistd.h>
#include <string.h>
#include <stdlib.h>
#include <stdio.h>
#include <errno.h>
#define MAX_LINE 16384
void do_read(evutil_socket_t fd, short events, void *arg);
void do_write(evutil_socket_t fd, short events, void *arg);
char
rot13_char(char c)
{
return c;
/* We don't want to use isalpha here; setting the locale would change
* which characters are considered alphabetical. */
if ((c >= 'a' && c <= 'm') || (c >= 'A' && c <= 'M'))
return c + 13;
else if ((c >= 'n' && c <= 'z') || (c >= 'N' && c <= 'Z'))
return c - 13;
else
return c;
}
struct fd_state {
char buffer[MAX_LINE];
size_t buffer_used;
size_t n_written;
size_t write_upto;
struct event *read_event;
struct event *write_event;
};
struct fd_state *
alloc_fd_state(struct event_base *base, evutil_socket_t fd)
{
struct fd_state *state = malloc(sizeof(struct fd_state));
if (!state)
return NULL;
state->read_event = event_new(base, fd, EV_READ|EV_PERSIST, do_read, state);
if (!state->read_event) {
free(state);
return NULL;
}
state->write_event =
event_new(base, fd, EV_WRITE|EV_PERSIST, do_write, state);
if (!state->write_event) {
event_free(state->read_event);
free(state);
return NULL;
}
state->buffer_used = state->n_written = state->write_upto = 0;
assert(state->write_event);
return state;
}
void
free_fd_state(struct fd_state *state)
{
event_free(state->read_event);
event_free(state->write_event);
free(state);
}
void
do_read(evutil_socket_t fd, short events, void *arg)
{
struct fd_state *state = arg;
char buf[1024];
int i;
ssize_t result;
while (1) {
assert(state->write_event);
result = recv(fd, buf, sizeof(buf), 0);
if (result <= 0)
break;
for (i=0; i < result; ++i) {
if (state->buffer_used < sizeof(state->buffer))
state->buffer[state->buffer_used++] = rot13_char(buf[i]);
if (buf[i] == '\n') {
assert(state->write_event);
**event_add(state->write_event, NULL);**
state->write_upto = state->buffer_used;
}
}
}
if (result == 0) {
free_fd_state(state);
} else if (result < 0) {
if (errno == EAGAIN) // XXXX use evutil macro
return;
perror("recv");
free_fd_state(state);
}
}
void
**do_write(evutil_socket_t fd, short events, void *arg)**
{
struct fd_state *state = arg;
while (state->n_written < state->write_upto) {
ssize_t result = send(fd, state->buffer + state->n_written,
state->write_upto - state->n_written, 0);
if (result < 0) {
if (errno == EAGAIN) // XXX use evutil macro
return;
free_fd_state(state);
return;
}
assert(result != 0);
state->n_written += result;
}
if (state->n_written == state->buffer_used)
state->n_written = state->write_upto = state->buffer_used = 1;
event_del(state->write_event);
}
void
do_accept(evutil_socket_t listener, short event, void *arg)
{
struct event_base *base = arg;
struct sockaddr_storage ss;
socklen_t slen = sizeof(ss);
int fd = accept(listener, (struct sockaddr*)&ss, &slen);
if (fd < 0) { // XXXX eagain??
perror("accept");
} else if (fd > FD_SETSIZE) {
close(fd); // XXX replace all closes with EVUTIL_CLOSESOCKET */
} else {
struct fd_state *state;
evutil_make_socket_nonblocking(fd);
state = alloc_fd_state(base, fd);
assert(state); /*XXX err*/
assert(state->write_event);
event_add(state->read_event, NULL);
}
}
void
run(void)
{
evutil_socket_t listener;
struct sockaddr_in sin;
struct event_base *base;
struct event *listener_event;
base = event_base_new();
if (!base)
return; /*XXXerr*/
sin.sin_family = AF_INET;
sin.sin_addr.s_addr = 0;
sin.sin_port = htons(40713);
listener = socket(AF_INET, SOCK_STREAM, 0);
evutil_make_socket_nonblocking(listener);
#ifndef WIN32
{
int one = 1;
setsockopt(listener, SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one));
}
#endif
if (bind(listener, (struct sockaddr*)&sin, sizeof(sin)) < 0) {
perror("bind");
return;
}
if (listen(listener, 16)<0) {
perror("listen");
return;
}
listener_event = event_new(base, listener, EV_READ|EV_PERSIST, do_accept, (void*)base);
/*XXX check it */
event_add(listener_event, NULL);
event_base_dispatch(base);
}
int
main(int c, char **v)
{
setvbuf(stdout, NULL, _IONBF, 0);
run();
return 0;
}
I'm not sure if I'm answering the same question you asked - I understand it as:
How does calling event_add(state->write_event, NULL) in do_read() lead to do_write() being invoked?
The key to figuring this out is understanding what the do_read() function is actually doing. do_read() is a callback function associated with a socket which has data to be read: this is set up with allocate_fd_state():
struct fd_state *
alloc_fd_state(struct event_base *base, evutil_socket_t fd)
{
/*
* Allocate a new fd_state structure, which will hold our read and write events
* /
struct fd_state *state = malloc(sizeof(struct fd_state));
[...]
/*
* Initialize a read event on the given file descriptor: associate the event with
* the given base, and set up the do_read callback to be invoked whenever
* data is available to be read on the file descriptor.
* /
state->read_event = event_new(base, fd, EV_READ|EV_PERSIST, do_read, state);
[...]
/*
* Set up another event on the same file descriptor and base, which invoked the
* do_write callback anytime the file descriptor is ready to be written to.
*/
state->write_event =
event_new(base, fd, EV_WRITE|EV_PERSIST, do_write, state);
[...]
return state;
}
At this point, though, neither of these events have been event_add()'ed to the event_base base. The instructions for what to do are all written out, but no one is looking at them. So how does anything get read? state->read_event is event_add()'ed to the base after an incoming connection is made. Look at do_accept():
void
do_accept(evutil_socket_t listener, short event, void *arg)
{
[ ... accept a new connection and give it a file descriptor fd ... ]
/*
* If the file descriptor is invalid, close it.
*/
if (fd < 0) { // XXXX eagain??
perror("accept");
} else if (fd > FD_SETSIZE) {
close(fd); // XXX replace all closes with EVUTIL_CLOSESOCKET */
/*
* Otherwise, if the connection was successfully accepted...
*/
} else {
[ ... allocate a new fd_state structure, and make the file descriptor non-blocking ...]
/*
* Here's where the magic happens. The read_event created back in alloc_fd_state()
* is finally added to the base associated with it.
*/
event_add(state->read_event, NULL);
}
}
So right after accepting a new connection, the program tells libevent to wait until there's data available on the connection, and then run the do_read() callback. At this point, it's still impossible for do_write() to be called. It needs to be event_add()'ed. This happens in do_read():
void
do_read(evutil_socket_t fd, short events, void *arg)
{
/* Create a temporary buffer to receive some data */
char buf[1024];
while (1) {
[ ... Receive the data, copying it into buf ... ]
[ ... if there is no more data to receive, or there was an error, exit this loop... ]
[ ... else, result = number of bytes received ... ]
for (i=0; i < result; ++i) {
[ ... if there's room in the buffer, copy in the rot13() encoded
version of the received data ... ]
/*
* Boom, headshot. If we've reached the end of the incoming data
* (assumed to be a newline), then ...
*/
if (buf[i] == '\n') {
[...]
/*
* Have libevent start monitoring the write_event, which calls do_write
* as soon as the file descriptor is ready to be written to.
*/
event_add(state->write_event, NULL);
[...]
}
}
}
[...]
}
So, after reading in some data from a file descriptor, the program starts waiting until
the file descriptor is ready to be written to, and then invokes do_write(). Program
flow looks like this:
[ set up an event_base and start waiting for events ]
[ if someone tries to connect ]
[ accept the connection ]
[ ... wait until there is data to read on the connection ... ]
[ read in data from the connection until there is no more left ]
[ ....wait until the connection is ready to be written to ... ]
[ write out our rot13() encoded response ]
I hope that a) that was the correct interpretation of your question, and b) this was a helpful answer.

Resources