php - Key-value storage for counters with ability of fast dumping data -

let's imagine that, have 300k keys simple counters in our storage counter key names, example: counter1_2014-03-25_00:01 counter2_2014-03-25_00:01 counter3_2014-03-25_00:01

each counter collecting data 1 minute, 1 key per 1 minute: counter1_2014-03-25_00:01, counter1_2014-03-25_00:02, counter1_2014-03-25_00:03 etc.

every next minute have dump counters previous minute , remove them storage. current implementation really simple , using redis hashes (hincr, hgetall, del).

//incrementing counter $redis->hincr('counters_2014-03-25_00:02', 'counter1');

//dumping , removing $result = $redis->multi() ->hgetall('counters_2014-03-25_00:02') //dumping ->del('counters_2014-03-25_00:02') //removing hash ->exec()

everything fine redis, dumping hgetall getting , slow large number of counters because hgetall have o(n) complexity.

right i'm looking approach allow:

• atomically incremented counters doing fast redis do
• quickly dump counter values collected previous minute
• remove counters collected previous minute storage
• no 100% persistence required, great plus

update: fast dumping required because need transfer locally collected data backend servers master server.

one solution can create simple keys, increment using incr , each time create key set expiry time using expire.