From FBSD_tips
C1t1zenJ0hn: added too many groups to his UID and couldn't log in, getting this: Dec 12 14:32:22 sshd[90838]: fatal: initgroups: : Invalid argument
dho: found this :
dho: [EINVAL] The number specified in the ngroups argument is larger
dho: than the NGROUPS limit.
dho: sys/sys/syslimits.h:#define NGROUPS_MAX 16 /* max supplemental group id's */
comperr ran out of ashared memory runninf VLC Gdk-WARNING **: shmget failed: error 12 (Cannot allocate memory)
eyecure noted :
comperr; sysctl -a |grep kern.ipc
(19:59:42) eyecue: take a squiz at the sh* or shm* values
(19:59:52) eyecue: shmall and shmmax for starters
(20:00:03) eyecue: that limits shared memory totals, among other things
and
http://www.postgresql.org/docs/8.2/static/kernel-resources.html has some details that describe what each one is
dont worry about the 'reasonable values' column, since theyre postgresql specific
- DDOS attack and suggestion from ##freebsd
(00:59:04) Kage`: I need emergency help setting up ipfw to block an incoming botnet on one of my jails
(00:59:41) Kage`: I don't know anything about ipfw and don't have any time to RTFM. I need emergency help, please.
(00:59:52) reppie: http://freebsd.org/handbook
(01:00:51) Detritus: If it's a botnet you are being attacked by many multiple IP's, that's nearly impossible to target a rule for.
(01:02:14) Kage`: Detritus, Well I need to do something NOW!
(01:02:18) Kage`: So can you help me?
(01:02:33) Kage`: They're all sending the same request with no headers
(01:02:37) cryptnix left the room (quit: Read error: 104 (Connection reset by peer)).
(01:02:39) Kage`: 165.21.155.11 - - [18/Dec/2007:05:49:18 +0000] "POST /pages/index/index.php HTTP/1.1" 200 4647 "-" "2756"
(01:02:39) Kage`: 41.234.166.248 - - [18/Dec/2007:05:51:25 +0000] "POST /pages/index/index.php HTTP/1.1" 200 4647 "-" "2320"
(01:02:39) Kage`: 72.234.219.215 - - [18/Dec/2007:05:51:25 +0000] "POST /pages/index/index.php HTTP/1.1" 200 4647 "-" "3968"
(01:02:41) Kage`: Etc.
(01:03:31) Kage`: reppie, if you've got nothing good to contribute, then kindly shut the fuck up.
(01:03:39) Kage`: Detritus, can you help me, please?
(01:03:58) Helio: 'don't feed the trolls'
(01:04:32) Detritus: I just said it's not possible to design a rule to block the requests from multiple random IP's. Even if you don't respond to requests it's still eating your bandwith. As i said. There is very little that can be done.
(01:04:43) Kage`: Well, what can I do?
(01:07:33) Detritus: Well you could switch your network provider to comcast. I understand that if they think you are using to many network resources they artifically inject IP resets into open connections.... That ought to help.
(01:08:42) gongoputch: a smart botnet might ignore the RST
(01:09:02) Detritus: I was being sarcastic anyway
(01:09:14) rumbleca: kage can you just shut its route off for now?
(01:09:24) Kage`: rumbleca, yeah, I can null the IP
(01:09:32) Kage`: But then I lose webroot connectivity
(01:09:35) lkthomas: route the ip address to 0.0.0.0
(01:09:46) rumbleca: Kage`: alllow from one ip, then block everything else
(01:09:48) Kage`: I want to just filter this shit
(01:09:50) lkthomas: usign script to block them ?
(01:09:53) gongoputch: blackhole the IPs as you identify the requests
(01:09:59) rumbleca: yeah
(01:10:02) rumbleca: that's a good one
(01:10:09) Kage`: gongoputch, ?
(01:10:28) gongoputch: you say that there are no headers?
(01:10:39) Kage`: gongoputch, just random integers or hashes
(01:10:48) Kage`: 165.21.155.11 - - [18/Dec/2007:05:49:18 +0000] "POST /pages/index/index.php HTTP/1.1" 200 4647 "-" "2756"
(01:10:53) Kage`: That's essentially what's sent
(01:10:54) rumbleca: this is port 80, right?
(01:10:56) Kage`: Yes
(01:11:02) gongoputch: no identifiable pattern?
(01:11:15) Kage`: gongoputch, they're all POSTing to /pages/index/index.php
(01:11:29) lkthomas: Kage`: any way to block POST request on your apache ?
(01:11:40) Kage`: Not without totally killing functionality
(01:11:42) gongoputch: is /pages/index/index.php essential?
(01:11:49) Kage`: gongoputch, it's the main site page
(01:12:10) lkthomas: so there have legal operating on POST to index.php as well
(01:12:19) Kage`: yes
(01:12:33) Kage`: Wait, hold on
(01:12:35) rumbleca: Kage`: are they going fast? there might be a way with ipfw limt
(01:12:36) lkthomas: can you just leave it until the raid is over ?
(01:12:38) rumbleca: err limit
(01:12:44) gongoputch: do the bots set REFERER header?
(01:12:44) Kage`: rumbleca, oh yes
(01:12:53) Kage`: rumbleca, about 2000 requests per second
(01:13:00) lkthomas: HOLY SHIT, LOL
(01:13:03) lkthomas: limit it
(01:13:07) lkthomas: limit the request
(01:13:10) Kage`: lkthomas, I'm estimating it's a 900+ botnet
(01:13:12) lkthomas: when it is over the limit, drop packet
(01:13:12) gongoputch: use PF's limits per state function
(01:13:20) lkthomas: agree with gongoputch
(01:13:27) Kage`: gongoputch, I don't know shit about ipfw. I need you to guide me please
(01:13:38) gongoputch: it is PF not IPFW
(01:13:46) Kage`: I don't know that either
(01:13:50) Kage`: I only know iptables :/
(01:13:51) gongoputch: and I am not good with PF, I am an IPFW ype
(01:13:57) Kage`: I'm new to bsd
(01:13:58) rumbleca: can you do something like limit src-addr 2
(01:14:01) rumbleca: on port 80?
(01:14:05) overrider: yes
(01:14:07) Kage`: rumbleca, ...how?
(01:14:23) gongoputch: do you have console access to the box?
(01:14:26) Kage`: Yes
(01:14:30) Kage`: I have full root to the host OS
(01:14:34) overrider: while im fighting with pf myself for a few days now, -> http://www.openbsd.org/faq/pf/ <- is helpful
(01:14:36) Kage`: It's one of the jails that's being attacked
(01:14:39) Kage`: Our webroot jail
(01:14:42) rumbleca: Kage`: check here: http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/firewalls-ipfw.html
(01:14:49) rumbleca: at the bottom, look for limit src-addr
(01:14:53) Kage`: I don't have time to rtfm >_<
(01:14:54) gongoputch: good, messing with networking over the network is .... frought with mishaps
(01:14:58) rumbleca: that is pretty crappy, but perhaps the best ipfw has to offer
(01:14:59) ***Kage` sighs
(01:15:05) rumbleca: $cmd 00400 allow tcp from any to me 80 in via $pif setup limit src-addr 2
(01:15:10) ***etherr buuummm !! Bang !! buf !! zbang !
(01:15:16) Kage`: I need this botnet to be quelled and my site brought back up like, 2 hours ago
(01:15:36) rumbleca: start using pf :o
(01:15:55) lkthomas: Kage`: you should use pf for advance firewalling
(01:16:07) Kage`: lkthomas, can you help me get at least this limit thing setup?
(01:16:19) lkthomas: in the same time, I don't know why freebsd wouldn't set ipfw still default installed but not pf
(01:16:25) rumbleca: Kage`: did you see that line I posted?
(01:16:27) lkthomas: Kage`: try rumbleca solution
(01:16:31) overrider: it will help you not much if you are attacked by a "botnet" , if there are enough people attacking you, and you cant control the router upstream of your webserver, and you are static ip, then you are hooped
(01:16:40) rumbleca: yeah
(01:16:41) overrider: could only try to connect to your isp
(01:16:46) Kage`: Didn't you see me say I don't have time to RTFM?
(01:16:48) Caelian|work__ left the room (quit: Read error: 110 (Connection timed out)).
(01:16:48) open_source_nut [n=osn@unaffiliated/opensourcenut/x-000001] entered the room.
(01:16:51) overrider: customer service that is
(01:16:53) Kage`: This downtime is costing me
(01:17:05) cl0ud: time to work with line providers
(01:17:13) Kage`: I'll rtfm later once I have some site functionality returned
(01:17:26) cl0ud: then call your ISP
(01:17:28) overrider: nothing you can do my man, if the attack is sufficient to fill your line, the only thing you can do is to contact your isp
(01:17:47) gongoputch: this is the sad fact of a DDOS
(01:17:51) Kage`: overrider, it's not really filling the line
(01:17:54) rumbleca: the easiest thing is to add that line I pasted to your config, that will help very minimally
(01:18:10) gongoputch: it doesn't matter what the poor host downstream does, if the pipe is filled
(01:18:11) Kage`: I'm on 10mbps burstable, and right now it's bursted to 100mbps, and I'm hitting about 50 on average
(01:18:44) rumbleca: its tying up his webserver that's all
(01:18:45) gongoputch: do you have a firewall loaded?
(01:18:45) Kage`: I did hit the full 100 earlier, but it capped then fell
(01:18:49) overrider: so it is hitting on the same port?
(01:18:54) Kage`: overrider, yes
(01:19:19) Kage`: gongoputch, Honestly, I don't know what my friend loaded this morning. He SAID he loaded ipfw
(01:19:23) Kage`: But when I do ipfw list
(01:19:33) gongoputch: do ipfw show
(01:19:39) Kage`: [root@nub /home/kage]# ipfw list
(01:19:39) Kage`: ipfw: getsockopt(IP_FW_GET): Protocol not available
(01:19:46) gongoputch: it is not loaded
(01:19:47) Kage`: [root@nub /home/kage]# ipfw show
(01:19:47) Kage`: ipfw: getsockopt(IP_FW_GET): Protocol not available
(01:19:53) gongoputch: loading it will lock you out
(01:19:56) Kage`: Yeah, so he didn't do the shit I asked him to >:/
(01:19:56) opensourcenut left the room (quit: Read error: 113 (No route to host)).
(01:20:19) gongoputch: so if you want to use rumbleca's rule, you better have console access
(01:20:46) Kage`: gongoputch, local?
(01:20:48) Kage`: No
(01:20:49) Detritus: or be ready to go for a drive to the hosting facility
(01:20:52) rumbleca: its better if you can use f
(01:20:54) Kage`: The server's in Atlanta
(01:20:54) rumbleca: use pf
(01:20:57) Kage`: I'm a good 3 states away
(01:21:17) gongoputch: so you don't have out of bans console access?
(01:21:24) rumbleca: do you need anything for pf to work in 6.2?
(01:21:26) Kage`: Sorry?
(01:21:26) gongoputch: band
(01:21:29) overrider: yeah use pf
(01:21:38) Kage`: gongoputch, no, I don't have anything of that sort
(01:21:44) lkthomas: LOL
(01:21:50) gongoputch: you are in a tough spot
(01:21:54) cl0ud: haha @ lkthomas
(01:21:55) ***Kage` sighs
(01:21:59) Kage`: This fucking sucks.
(01:22:21) overrider: umm learn the tools before employing them...
(01:22:27) lkthomas: Kage`: I think you guys is kind of admin that config server without perpare for a raid
(01:22:33) gongoputch: cl0ud: please, huh?
(01:23:06) Kage`: lkthomas, english?
(01:23:18) rumbleca: Kage`: can you accept dcc? I sent you a file
(01:23:27) Kage`: rumbleca, and that file is?
(01:23:38) rumbleca: its a pf config file
(01:23:42) Kage`: Ok, send it
(01:24:15) gongoputch: is PF kld loadable?
(01:24:41) cl0ud: gongoputch, uhh ok?
(01:25:07) gongoputch: cl0ud: no need to laugh at the dude, ok?
(01:25:29) gongoputch: ah, so sorry
(01:25:34) gongoputch: I misread that
(01:25:37) cl0ud: gongoputch, I think I'm free to laugh WITH someone in a public channel whenever I want
(01:25:38) cvn: hi whats the command to check the ports that are already in used?
(01:25:42) cl0ud: gongoputch, :)
(01:25:57) gongoputch: yes, you certainly are
(01:26:13) gongoputch: up late and sick, I misread
(01:26:21) cl0ud: ah, bummer on the sickness
(01:26:25) umtc: cvn, you could use sockstat or netstat
(01:26:26) gongoputch: cvn: sockstat
(01:26:50) gongoputch: cl0ud: I'll get better ... soon ... or die :)
(01:27:01) cl0ud: gongoputch, that's the way it goes
(01:27:13) Helio: my baloney has a first name, it's s. h. i. t. e, my baloney has a second name, it's b. u. t. t. f. u. ... you see where I'm going with this
(01:27:32) gongoputch: I am interested to see if rumbleca's pf can help Kage`
(01:27:57) ritalin left the room (quit: Read error: 110 (Connection timed out)).
(01:28:47) overrider: pf is kld loadeable
(01:29:36) nighty^ left the room (quit: Remote closed the connection).
(01:30:22) proteusguy [n=proteusg@ppp-124.120.226.207.revip2.asianet.co.th] entered the room.
(01:30:52) Ci-Dev left the room (quit: ""Democracy is two wolves and a lamb voting on what to have for lunch. Liberty is a well armed lamb contesting the vote." -- B).
(01:32:40) rumbleca: I just sent it to him
(01:32:45) rumbleca: not sure if he will use it or not
(01:33:34) gongoputch: what does it look like?
(01:33:53) gongoputch: I am interested in learning PF
(01:34:25) Kage`: rumbleca, I will use it, but I'm trying to figure a way to do this without kvm
(01:34:31) Kage`: What about a cron?
(01:34:45) Kage`: Have it auto-kill pf every 60 seconds
(01:34:50) Kage`: Hmmm...
(01:34:56) gongoputch: does kld loading PF lock you out like IPFW?
(01:34:57) rumbleca: I can dcc it
(01:35:03) rumbleca: Kage`: yean
(01:35:12) rumbleca: Kage`: you can use cron, or at, and just do a pfctl -d
(01:35:23) gongoputch: rumbleca: DDC busted onmy client, could you pastebin?
(01:35:36) rumbleca: gongoputch: ok, but this is not something I am proud of :)
(01:35:38) overrider: gongoputch, it is possible to lock yourself out, yes :-)
(01:35:55) ***Kage` sighs and nulled the IP for tonight
(01:35:57) gongoputch: overrider: but is there a default to deny?
(01:36:09) Kage`: My host is going to implement a kvm system in the next 6 months :(
(01:36:13) ***Kage` sighs
(01:36:17) Kage`: I need a damn cigarette...
(01:36:27) overrider: gongoputch, no, it will read the /etc/pf.conf file when you enable it, which you want to customize before you enable of
(01:36:30) overrider: eh pf
(01:36:44) rumbleca: gongoputch: http://pastebin.ca/821615
(01:36:53) rumbleca: there are much better ways to do it, but that's mine, more or less
(01:37:40) gongoputch: rumbleca: ty
(01:38:12) overrider: rumbleca, you dont have a default deny statement on top, not good practice
(01:38:58) gongoputch: in IPFW deny usually comes last, this is because the order is different with PF?
(01:39:18) rumbleca: overrider: you mean a block in?
(01:39:24) overrider: pf.conf should start with a block log all, at least thats how i learned it
(01:39:33) vcygnus: Kage`, try to make the remote connection to another computer in the same LAN segment than that server, but not to the server directly, so if something goes wrong, at least you're still in the LAN
(01:39:50) rumbleca: overrider: that's in the pf faq, I should probably do that
(01:39:56) rumbleca: gongoputch: the order is important... all those sections
(01:39:58) overrider: yes
(01:40:31) lkthomas: oh my god
(01:40:36) lkthomas: problem still not fixed ?
(01:41:10) overrider: here is the one i use at home, really simple but a good start, at work its way more complicated though, managing traffic for 4 interfaces, but for home, this has worked for me and my wife -> http://pastebin.com/m164cdda3
(01:42:38) rumbleca: cool
(01:42:43) gongoputch: I built a virtual network lab (http://bsdtips.utcorp.net/mediawiki/index.php/Virtual_network_testbed) to test stuff like this, and fill in the table here http://bsdtips.utcorp.net/mediawiki/index.php/Firewall_chart
(01:42:59) Kwitschibo left the room (quit: Read error: 110 (Connection timed out)).
(01:43:45) gongoputch: boxes and cables and switches, etc. are such a bother
(01:43:47) rumbleca: wow, cool
(01:44:06) lkthomas: I got a question
(01:44:19) lkthomas: is there have any 1G NIC card which could last more than 1.5M packet per second loading ?\
(01:45:16) gongoputch: bbiab
(01:45:31) lkthomas: huh ?
(01:45:48) rumbleca: be back in a bit, heh
(01:46:01) rumbleca: no idea lkthomas, do you have a card that is failing miserably?
(01:46:24) lkthomas: I am looking for a card which could last when DDOS appear
(01:49:20) open_source_nut is now known as opensourcenut
(01:50:15) rumbleca: k, thanks override, I Added that
(01:50:17) rumbleca: plus antispoof
