Monday, January 17, 2022

Redis NoSQL schemaless Mutlidata type supporting Scalable, durable , recoverable Database

Redis is nosql database. It is also called as Remote dictionary server. Generally developer use the redis to use it as an intermediate between the application and original DB for cache first mechanism. But Redis itself is a memory first Key value type no sql data base. It support different type of data type like string, set, hash etc.
Let's take an example of social app. lets assume we had an requirement where we need to store the user information like name, add, telephone number, password etc. as a social app each use is connected to other user so we have graph DB like requirement. Additionally we also need to have quick search on this huge db like apache solr. Also this app has facilities to store the files i.e. document voice, video, txt file etc. All this can be done in one database and that is Redis. Redis come with different module. The core one is called Redis core which support all above datatype like string set has etc but it also comes with separate module to address issues like graph db, quick search, storing files etc. for this we have the following modules

1- Quick search – Redis search
2- Storing user informaiton – Redis time series DB.
3- Strong audio, video and files – Redis JSON
4- Graph DB- Redis graph.

SCALLING:-

In addition to this Redis has one big advantage of easy SCALLING in comparison of other nosql DB. It support scalling using two option

1- Clustering
2- Sharding

1- Clustering – In this we replicas the master Redis DB so that Master work as read-write db and rest other work as read only db. when master dies any one of the replica become master. In addition to this redis also support best option of back up. It has two options

a:- Timestamp :- in this concept we take the redis db back up at definite interval of time. only issues is if the last timestamp back up fail we get the date till the secondlast timestamp for recovery.
b:- AOF – append only file – in this we append the changes first in the file (backup) and then do the changes in redis. here we had runtime back up and no data is loss but this has some performance issues.

2-Sharding :- in this concept we divide our master data into different data set and store the data set itself on different machine or Node. This will also scale the machine also give us the data disaster recovery mechanism.

In redis We can use the both concept at the same time. To achieve this, modifies the redis.conf file and modify “appendonly” parameter from “no” to “yes”

To take the time stamp just execute the save option on the command prompt of redis cli tool it will create dump.rdb as shown below.

Back up:-

Now the question is how redis support the 100% back up. lets say if not only the node but the complete serveror M/C is down in that case we are using the remote storing concept i.e. if you are using Cloud the we will have Redis service and DB running EC2 and their back up is taken on Elastic block storage server. so in this case if the whole EC2 is down still we have the back up on Elastic.

Durability:-

Redis also support Durability :- for this it use the concept of Redis on flash. i.e. it will also use the SSD card to store the data in ram. Most frequent data will be stored in RAM and less will be stored in SSD.

There is also a concept called as active active geo distribution. Using this concept we can restore the whole data center. let say we have a data center in india and we have customer in UK and Canada for their speedy application use it is better to have the data nearer to them so taht latency will be less. This is achieved by having data center in canada and also in uk. but the issues is how to sync them all. this is also taken care by redis enterprise. so if one whole datacenter fail we can build it from taking the record from other datacenter

Now the question is in world of Kubernetes how to manage this Redis for scalability, storing, backup , durability etc. for this We have an Redis operator already build which take care of all this things i.e. deployment, configuring, scaling , backup, recovering and much more.

Now lets try to install Redis as we are using windows please follow below steps

Go to belwo sites and clone the repo.

https://github.com/zkteco-home/redis-windows

Install service instance:

redis-server –service-install redis.conf –loglevel verbose

Staring service
redis-server –service-start

Starting client

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
C:\redis-windows>redis-server --service-install redis.conf --loglevel verbose
[9192] 17 Jan 13:16:35.992 # Granting read/write access to 'NT AUTHORITY\NetworkService' on: "C:\redis-windows" "C:\redis-windows" "C:\redis-windows\"
[9192] 17 Jan 13:16:35.993 # Redis successfully installed as a service.
 
C:\redis-windows>redis-server --service-start
[7744] 17 Jan 13:17:11.154 # Redis service successfully started.
 
C:\redis-windows>redis-cli.exe
127.0.0.1:6379>
 
check connection using word ping we get pong
 
'-  checking connection is live
 
C:\redis-windows>redis-cli.exe
127.0.0.1:6379> ping
PONG
127.0.0.1:6379> echo "hello world"
"hello world"
 
'- Quitting connection
127.0.0.1:6379> quit
 
C:\redis-windows>
 
 
Now lets play with few of the basic concept of Redis.
 
'- setting values
 
127.0.0.1:6379> set foo siddhu
OK
127.0.0.1:6379> get foo
"siddhu"
 
'- incr and decr values.
 
 
127.0.0.1:6379> incr siddhu
(integer) 1
127.0.0.1:6379> incr siddhu
(integer) 2
127.0.0.1:6379> incr siddhu
(integer) 3
127.0.0.1:6379> get siddhu
"3"
127.0.0.1:6379> decr siddhu
(integer) 2
127.0.0.1:6379> decr siddhu
(integer) 1
127.0.0.1:6379> decr siddhu
(integer) 0
127.0.0.1:6379> decr siddhu
(integer) -1
127.0.0.1:6379>
 
'- To check if exist 1 means key exist and 0 means it does not exist i.e. we did not set the value of that key.
 
127.0.0.1:6379> exists foo
(integer) 1
127.0.0.1:6379> exists siddhu
(integer) 1
127.0.0.1:6379> exists test
(integer) 0
127.0.0.1:6379>
 
'- to delete the key use belwo command 1 means key is deleted and 0 means did not found the key.
 
127.0.0.1:6379>  del siddhu
(integer) 1
127.0.0.1:6379> del siddhu
(integer) 0
127.0.0.1:6379> get siddhu
(nil)
 
'- flushall to remove all the key that is set by use
127.0.0.1:6379> flushall
OK
127.0.0.1:6379>
 
 
'- Expire the key
 
127.0.0.1:6379> set siddhtime "hello time"
OK
127.0.0.1:6379> get siddhtime
"hello time"
127.0.0.1:6379> expire siddhtime 10
(integer) 1
127.0.0.1:6379> ttl
(error) ERR wrong number of arguments for 'ttl' command
127.0.0.1:6379> ttl siddhtime
(integer) 2
127.0.0.1:6379> ttl siddhtime
(integer) -2
127.0.0.1:6379> get siddhtime
(nil)
127.0.0.1:6379>
 
'- Setting expriration and value for the key in one go
 
127.0.0.1:6379> setex siddhutime 15 "hello"
OK
127.0.0.1:6379> get siddhutime
"hello"
127.0.0.1:6379> ttl siddhutime
(integer) 8
127.0.0.1:6379> get siddhutime
(nil)
127.0.0.1:6379>
 
'- persist to overcome expiration i.e. it will close the expiration clock of the key.
 
127.0.0.1:6379> SETEX siddhutime 180 "siddhu value"
OK
127.0.0.1:6379> persist siddhutime
(integer) 1
127.0.0.1:6379> get siddhutime
"siddhu value"
127.0.0.1:6379>
 
'- Multiple setting values using MSET
 
127.0.0.1:6379> mset siddhu1 "siddhu1 value" siddhu2 "siddhu2 value"
OK
127.0.0.1:6379> get siddhu1
"siddhu1 value"
127.0.0.1:6379> get siddhu2
"siddhu2 value"
127.0.0.1:6379>
 
 
'- Appending the value in key
127.0.0.1:6379> append siddhu1 "appending values"
(integer) 29
127.0.0.1:6379> get siddhu1
"siddhu1 valueappending values"
127.0.0.1:6379>
 
'- Rename the key.
 
127.0.0.1:6379> rename siddhu1 siddhunew1
OK
127.0.0.1:6379> get siddhu1
(nil)
127.0.0.1:6379> get siddhunew1
"siddhu1 valueappending values"
127.0.0.1:6379>
 
 
'- List datatype use lpush to add the value in the list and lrange  to fetch the value from it.
 
127.0.0.1:6379> lpush siddhulist element1
(integer) 1
127.0.0.1:6379> lpush siddhulist element2
(integer) 2
 
 
127.0.0.1:6379> lrange siddhulist 0 10
1) "element2"
2) "element1"
 
To see all the values use below command
127.0.0.1:6379> lrange siddhulist 0 -1
1) "element2"
2) "element1"
 
'- length of the list
 
127.0.0.1:6379> llen siddhulist
(integer) 2
 
'- Remove the item from the list from the top use
 
127.0.0.1:6379> RPUSH mylist "one" "two" "three" "four" "five"
(integer) 5
127.0.0.1:6379> lpop mylist 1
1) "one"
127.0.0.1:6379> lrange mylist 0 -1
1) "two"
2) "three"
3) "four"
4) "five"
127.0.0.1:6379> lpop mylist 1
1) "two"
127.0.0.1:6379> lrange mylist 0 -1
1) "three"
2) "four"
3) "five"
127.0.0.1:6379> lpop mylist 2
1) "three"
2) "four"
127.0.0.1:6379> lrange mylist 0 -1
1) "five"
 
Same way to remove from the bottom use RPOP
 
 
127.0.0.1:6379> rpop mylist
"five"
127.0.0.1:6379> rpop mylist
"four"
127.0.0.1:6379> lrange mylist 0 -1
1) "one"
2) "two"
3) "three"
127.0.0.1:6379>
 
'- Instert the value in between the key
 
127.0.0.1:6379> linsert mylist before two "newvalue"
(integer) 4
127.0.0.1:6379> lrange mylist 0 -1
1) "one"
2) "newvalue"
3) "two"
4) "three"
127.0.0.1:6379>
 
 
'- SET data type. Set do not allow the duplicate values. to add use sadd and to verify set has the key use sismember.
 
 
127.0.0.1:6379> sadd siddhuset siddhu1
(integer) 1
127.0.0.1:6379> sadd siddhuset siddhu2
(integer) 1
127.0.0.1:6379> sadd siddhuset siddhu3
(integer) 1
127.0.0.1:6379> sismember siddhuset 1
(integer) 0
127.0.0.1:6379> sismember siddhuset siddhu1
(integer) 1
127.0.0.1:6379> sismember siddhuset siddhu2
(integer) 1
127.0.0.1:6379> sismember siddhuset siddhu4
(integer) 0
127.0.0.1:6379>
 
'- to get all the memeber of the set use smembers commands.
 
127.0.0.1:6379> smembers siddhuset
1) "siddhu3"
2) "siddhu1"
3) "siddhu2"
127.0.0.1:6379>
 
'- To get the number of element from the set use below scard command
 
 
127.0.0.1:6379> scard siddhuset
(integer) 3
127.0.0.1:6379> smembers siddhuset
1) "siddhu3"
2) "siddhu1"
3) "siddhu2"
127.0.0.1:6379>
 
'- to find the diff between the two set use sdiff
 
127.0.0.1:6379> smembers siddhuset
1) "siddhu3"
2) "siddhu1"
3) "siddhu2"
127.0.0.1:6379> sadd siddhusetdiff siddhu1
(integer) 1
127.0.0.1:6379> sadd siddhusetdiff siddhu2
(integer) 1
127.0.0.1:6379> sadd siddhusetdiff siddhu4
(integer) 1
127.0.0.1:6379> smembers siddhuset
1) "siddhu3"
2) "siddhu1"
3) "siddhu2"
127.0.0.1:6379> sdiff siddhusetdiff siddhuset
1) "siddhu4"
 
'- To move the value to the other set use smove
 
127.0.0.1:6379> smove siddhuset siddhusetdiff siddhu3
(integer) 1
127.0.0.1:6379> smembers siddhusetdiff
1) "siddhu3"
2) "siddhu1"
3) "siddhu4"
4) "siddhu2"
127.0.0.1:6379>
 
'- To remove element use srem
 
127.0.0.1:6379> srem siddhusetdiff siddhu3
(integer) 1
127.0.0.1:6379> smembers siddhusetdiff
1) "siddhu1"
2) "siddhu4"
3) "siddhu2"
 
'- Sorted set is similar to set  but it have score that we assign to the value. And this score will be used for sorting them in increment or decrement.
 
127.0.0.1:6379> zadd myzset 1 "one" 2 "two" 2 "TWO" 3 "three" 4 "four"
(integer) 5
127.0.0.1:6379> zrange myzset 0 -1
1) "one"
2) "TWO"
3) "two"
4) "three"
5) "four"
 
'- we can also give the decimal value and string for sorting.
 
127.0.0.1:6379> zadd myzset 1.5 "one half" 2.5 "twohalf"
(integer) 2
127.0.0.1:6379> zrange myzset 0 -1
1) "one"
2) "one half"
3) "TWO"
4) "two"
5) "twohalf"
6) "three"
7) "four"
127.0.0.1:6379> zadd myzset 1.3 "one three" 2.3 "two three"
(integer) 2
127.0.0.1:6379> zrange myzset 0 -1
1) "one"
2) "one three"
3) "one half"
4) "TWO"
5) "two"
6) "two three"
7) "twohalf"
8) "three"
9) "four"
 
127.0.0.1:6379> zadd myzset "1.02" "one zero two"
(integer) 1
127.0.0.1:6379> zadd myzset "1.01" "one zero one"
(integer) 1
127.0.0.1:6379> zadd myzset "1.02" "one zero two"
(integer) 0
127.0.0.1:6379> zrange myzset 0 -1
 1) "one"
 2) "one zero one"
 3) "one zero two"
 4) "one two"
 5) "one three"
 6) "one half"
 7) "TWO"
 8) "two"
 9) "two three"
10) "twohalf"
11) "three"
12) "four"
 
'- to find the rank in sorted set use below command
127.0.0.1:6379> zrank myzset one
(integer) 0
127.0.0.1:6379> zrank myzset "one zero one"
(integer) 1
127.0.0.1:6379>
 
'- to increment rank of score use below ZINCRBY command.
 
127.0.0.1:6379> zrange myzset 0 -1
 1) "one"
 2) "one zero one"
 3) "one zero two"
 4) "one two"
 5) "one three"
 6) "one half"
 7) "TWO"
 8) "two"
 9) "two three"
10) "twohalf"
11) "three"
12) "four"
127.0.0.1:6379> ZINCRBY myzset 2 "one"
"3"
127.0.0.1:6379> zrange myzset 0 -1
 1) "one zero one"
 2) "one zero two"
 3) "one two"
 4) "one three"
 5) "one half"
 6) "TWO"
 7) "two"
 8) "two three"
 9) "twohalf"
10) "one"
11) "three"
12) "four"
 
 
'- Hashset to be used to set the key value in string and get the same as shown below.
 
127.0.0.1:6379> HSET siddhuhash name siddhu
(integer) 1
127.0.0.1:6379> HSET siddhuhash email siddhu@gmail.com
(integer) 1
127.0.0.1:6379> hget siddhuhash name
"siddhu"
127.0.0.1:6379> hget siddhuhash email
"siddhu@gmail.com"
127.0.0.1:6379>
 
 
'- to get all from hashset use below command
 
127.0.0.1:6379> hgetall siddhuhash
1) "name"
2) "siddhu"
3) "email"
4) "siddhu@gmail.com"
 
'- for length use hlen
127.0.0.1:6379> hlen siddhuhash
(integer) 2
 
'- to get only key from the hashset use hkeys
 
127.0.0.1:6379> hkeys siddhuhash
1) "name"
2) "email"
127.0.0.1:6379>
 
 
 
'- to get only values from the hashset use hvals
 
127.0.0.1:6379> hvals siddhuhash
1) "siddhu"
2) "siddhu@gmail.com"
127.0.0.1:6379>
 
 
'- to set multiple value in hashset in one command
 
127.0.0.1:6379> hmset siddhunewhash name "siddhu dhumale" age "43" country "india"
OK
127.0.0.1:6379> hgetall siddhunewhash
1) "name"
2) "siddhu dhumale"
3) "age"
4) "43"
5) "country"
6) "india"
127.0.0.1:6379>

These are the data types supported by Redis

Bitmaps
Cluster
Connection
Geo
Hashes
HyperLogLog
Keys
Lists
Pub/Sub
Scripting
Sentinel
Server
Sets
Sorted Sets
Streams
Strings
Transactions

LHS-BS-HTC-GPSC
list, hash, string, Bitmap, set, sortedset, hyperloglog, transection, connection , Geo, pub/sub, server, streams, sentinel, scripting, Cluster

Here is the java maven project setup for working with Redis using lettuce-core jar.

https://lettuce.io/

Pom.xml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
    <modelVersion>4.0.0</modelVersion>
    <groupId>siddhu.redis</groupId>
    <artifactId>siddhu-redis</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>SiddhuRedis</name>
    <description>This is simple Redis example</description>
    <dependencies>
        <dependency>
            <groupId>io.lettuce</groupId>
            <artifactId>lettuce-core</artifactId>
            <version>6.1.6.RELEASE</version>
        </dependency>
    </dependencies>  
</project>

SiddhuRedis.java

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
/**
 *
 */
package com.siddhu;
 
import io.lettuce.core.RedisClient;
import io.lettuce.core.api.StatefulRedisConnection;
import io.lettuce.core.api.sync.RedisCommands;
 
/**
 * @author Siddhartha
 *
 */
public class SiddhuRedis {
 
    /**
     *
     */
    public SiddhuRedis() {
        // TODO Auto-generated constructor stub
    }
 
    /**
     * @param args
     */
    public static void main(String[] args) {
        // TODO Auto-generated method stub
        RedisClient redisClient = RedisClient.create("redis://password@localhost:6379/0");
        StatefulRedisConnection<String, String> connection = redisClient.connect();
        RedisCommands<String, String> syncCommands = connection.sync();
 
        syncCommands.set("key", "Hello, Redis!");
        syncCommands.hset("siddhuhash1", "siddhuhash2", "siddhuhash3");
         
        System.out.println("syncCommands.hget:"+syncCommands.hget("siddhuhash1", "siddhuhash2"));
        System.out.println("key:"+syncCommands.keys("*"));
        connection.close();
        redisClient.shutdown();
 
    }
 
}

No comments: